DeepSeek-V3 ☆

DeepSeek-V3 (December 2024) is a 671B-parameter MoE activating 37B per token, trained on 14.8T tokens across 2,048 H800 GPUs with a disclosed final training-run cost of about $5.576M (excluding research and prior runs — a caveat often dropped in retellings). Multi-head latent attention, FP8 training and multi-token prediction made it the most-studied efficiency result in the field; weights are open. V3.1 (August 2025) added hybrid thinking; V3.2-Exp cut long-context costs with sparse attention.