{"data":{"slug":"training","etype":"concept","name":"Training & Pre-training","cat":"Training & Inference","subtitle":"Months of thousands of GPUs predicting the next token, trillions of times.","summary":"Pre-training runs a model over trillions of tokens, nudging weights after every prediction error, in synchronized runs across tens of thousands of accelerators — the workload the biggest clusters on this site were built for. A frontier run costs hundreds of millions of dollars in compute, fails in exotic ways (silent data corruption, stragglers, loss spikes), and is why checkpointing, interconnects and reliability engineering are front-page infrastructure topics.","attrs":null,"kf":null,"aliases":["pre-training","pretraining"],"ticker":null,"exchange":null,"hq":null,"founded":null,"website":null,"status":null,"region":null,"asof":"2026-01-15","sources":[{"url":"https://ai.meta.com/research/publications/the-llama-3-herd-of-models/","label":"Llama 3 herd paper (training at scale)"}]},"meta":{"found":true,"count":1,"served_from":"site cache (no live upstream calls)","as_of":"2026-10-14T05:44:18+00:00","disclaimer":"Research and information only — not investment advice. Structured records carry basis labels (vendor spec / reported / estimate); market quotes are delayed/indicative.","attribution":"Records compiled from vendor documentation, official filings and reputable public reporting; market quotes via financial data providers. See /data-sources."}}