The Pile ☆

The Pile (2020) mixed 22 sources — papers, code, books, forums — and showed curation quality mattered as much as scale. It trained the GPT-Neo/GPT-J lineage that kept open research alive, and its inclusion of the Books3 shadow-library became a defining copyright battleground.