The Pile
Dataset
Curated Corpus
EleutherAI's 825 GB of curated diversity — open-source pre-training's foundation.
You're looking at a dataset record: a training-data collection — what's in it and which models learned from it.
The Pile (2020) mixed 22 sources — papers, code, books, forums — and showed curation quality mattered as much as scale. It trained the GPT-Neo/GPT-J lineage that kept open research alive, and its inclusion of the Books3 shadow-library became a defining copyright battleground.
How this connects to the rest of the stack
This record's links at a glance — click a group to jump to it.
Related
Sources
Record reviewed as of 2026-01-15 — see Methodology.