A frontier training run is less a computation than an industrial project — months long, hundreds of millions of dollars, with its own logistics, failure modes and night shifts.
First, the data
Teams assemble tens of trillions of tokens: filtered web crawl (Common Crawl lineage, FineWeb-class processing), code, licensed corpora, and — increasingly — synthetic data generated by earlier models. Deduplication, quality scoring and contamination checks (keeping benchmark answers out) consume real compute before training even starts. Data mix is guarded like a recipe because, per Chinchilla, it is half the result.
Split the model, split the data, split everything
No chip holds a frontier model. Training shards it three ways at once — layers across pipeline stages, matrices across tensor-parallel groups, batches across data-parallel replicas — the '3D parallelism' packaged in Megatron and DeepSpeed. MoE adds expert parallelism. The cluster's network then synchronizes gradients every step — which is why fabric quality, not FLOPS, caps utilization ('MFU', typically 35–45% at frontier scale on disclosed runs).
The unglamorous heroics: keeping it alive
At 100,000 GPUs, something is always dying. Meta's Llama 3 paper disclosed the texture: hundreds of unplanned interruptions over a 54-day run, mostly hardware — failed GPUs, HBM errors, cables. Silent data corruption (a chip miscalculating without erroring) is the nightmare case. The defenses are checkpointing to storage every few minutes, automatic straggler detection, spare capacity hot and waiting, and loss-curve babysitting around the clock — a 'loss spike' at 3 a.m. means humans deciding whether to rewind.
Then the second training
The pre-trained artifact is a wild predictor, not a product. Post-training — instruction tuning, RLHF, and RL on verifiable tasks for reasoning — consumes a growing fraction of total compute and most of the differentiation. Labs describe frontier post-training as its own multi-week, multi-thousand-GPU campaign with human data operations (Scale-class vendors) attached.
Ship it
Evaluation gauntlets (benchmarks + internal safety evals), red-teaming, quantized serving builds, and a staged rollout. Total elapsed time from data freeze to launch: typically two to four quarters — which, note, is longer than the hardware generation cadence chasing it.
10
