Inference
Running the model — now the majority of AI compute and the focus of the buildout.
You're looking at a concept explainer: what this term means in plain English, and the records it ties together.
Inference is every ChatGPT answer, Copilot completion and API call — and as products scaled, it overtook training as the dominant AI workload. It splits into prefill (reading the prompt, compute-bound) and decode (generating tokens, memory-bandwidth-bound), which is why HBM decides serving economics, why engines like vLLM matter, and why hardware now specializes by phase (Rubin CPX). 'Tokens per dollar per watt' is the metric the whole industry is converging on.
How this connects to the rest of the stack
This record's links at a glance — click a group to jump to it.
Related
Sources
Record reviewed as of 2026-01-15 — see Methodology.