Training & Pre-training
概念
训练与推理
Months of thousands of GPUs predicting the next token, trillions of times.
您正在查看一条概念解释记录：该术语的通俗含义，以及它所关联的各条目。
Pre-training runs a model over trillions of tokens, nudging weights after every prediction error, in synchronized runs across tens of thousands of accelerators — the workload the biggest clusters on this site were built for. A frontier run costs hundreds of millions of dollars in compute, fails in exotic ways (silent data corruption, stragglers, loss spikes), and is why checkpointing, interconnects and reliability engineering are front-page infrastructure topics.
它如何与整个技术栈相连
本记录的关联链接一览——点击分组可直接跳转。
来源
Llama 3 herd paper (training at scale) ↗
记录审阅截至 2026-01-15 — 查看 方法论.