The records behind this event
1-
Research Paper
Training Compute-Optimal LLMs: Chinchilla (2022)
DeepMind's correction: most models were undertrained on data.
Hoffmann et al. showed compute-optimal training wants ~20 tokens per parameter, far more data than the field was using, so a 70B model trained right beat a 280B one. It reshaped every subsequent training run and made data supply a first-order constraint.
Open record →