FlashAttention (2022) ☆

Tri Dao's FlashAttention reordered attention computation around GPU memory hierarchy — exact results, several times faster, memory linear instead of quadratic. Versions 2 and 3 chase each hardware generation; virtually every serious training and inference stack ships it. Systems research as leverage: one algorithm changed the economics of every cluster.