Between your Enter key and the first word back, a small factory runs. Serving engineers call the two phases prefill and decode, and the distinction organizes the entire inference industry.
Prefill: read everything at once
The model ingests your whole prompt in parallel — one big matrix pass, compute-bound, the GPU briefly saturated. Long contexts make this expensive: a 100K-token prompt is real work before any answer exists (hence prefill-specialist hardware like Rubin CPX). Prefill produces the first token and, as a by-product, the KV cache — stored attention state for every prompt token.
Decode: one token at a time
Generation is sequential: predict a token, append, repeat. Each step must stream the model's active weights from HBM, so decode is memory-bandwidth-bound — the GPU's compute units mostly wait. This is why bandwidth headlines every spec sheet and why the KV cache matters: it grows with context length and batch size until memory, not compute, caps how many users a GPU serves.
The tricks that make it affordable
Continuous batching — dozens of conversations share each weight-streaming pass (vLLM's PagedAttention made the memory bookkeeping efficient; SGLang adds prefix reuse). Quantization — FP8/FP4 weights halve and re-halve the bytes per token. Speculative decoding — a small draft model guesses several tokens, the big model verifies in one pass. Disaggregation — separate prefill and decode fleets (NVIDIA Dynamo, DeepSeek's serving stack) so each phase gets the hardware it wants. Together these took open-model serving costs down orders of magnitude in three years.
Reasoning changed the bill
Thinking models generate thousands of hidden tokens before answering — decode-heavy, cache-hungry, exactly the expensive phase. The 'age of inference' buildout (Ironwood, B300's 288 GB, the memory arms race) is downstream of this one product decision: models that think.
