Every AI workload is one of two things, and they want different machines, different facilities, even different power deals.
Training: a months-long moonshot
Training a frontier model is one enormous synchronized computation: tens of thousands of accelerators working in lockstep for weeks or months, all of them stalling if one hiccups. It wants maximum interconnect bandwidth (NVLink domains, fat fabrics), sits comfortably in one giant remote campus (latency to users is irrelevant), and tolerates almost no failures — a fault at hour 700 that isn't checkpointed is money on fire. Cost is capitalized: Llama 3.1 405B consumed about 39 million H100-hours (disclosed); frontier runs are estimated in the hundreds of millions of dollars of compute.
Inference: a forever utility
Inference is the opposite shape: millions of small independent requests, every second, forever. It scales horizontally (any number of 8-GPU islands), cares about latency (so it spreads geographically toward users), and its cost is operational — cents per million tokens, paid on every query. Its two phases even stress different resources: reading your prompt (prefill) is compute-bound; generating the answer (decode) is memory-bandwidth-bound. Hardware is now specializing by phase — see Rubin CPX.
The crossover
Through 2023 the buildout story was training clusters. By 2025, with ChatGPT near a billion weekly users and agents multiplying tokens per task, industry consensus put the majority of AI compute on inference — NVIDIA has described inference as the dominant workload driving Blackwell demand, and Google built its Ironwood TPU explicitly for the "age of inference."
Why it matters to everything else here
Reasoning models blurred the line — they spend training-scale compute at answer time. And the economics differ existentially: training costs are a bet repaid (or not) over a model's life; inference margin is the industry's actual business model. When analysts argue about the bubble question, they are mostly arguing about whether inference revenue grows into the training and infrastructure bill this site catalogs.
