DeepSpeed ☆

DeepSpeed's ZeRO optimizer sharding let models far larger than one GPU's memory train efficiently, powering the Megatron-Turing era and countless open efforts since. Its ideas (now echoed by PyTorch FSDP) defined how the industry splits models across thousands of devices.