Training Compute-Optimal LLMs — Chinchilla (2022) ☆

Hoffmann et al. showed compute-optimal training wants ~20 tokens per parameter — far more data than the field was using — so a 70B model trained right beat a 280B one. It reshaped every subsequent training run and made data supply a first-order constraint.