Competitors have matched or beaten NVIDIA silicon on specific specs for years — MI300X had more memory than H100, Gaudi undercut on price. The sockets mostly stayed NVIDIA anyway. The reason is CUDA: not one product but nineteen years of compounding software.
What the moat is made of
Layer one: kernels — cuDNN, cuBLAS and thousands of hand-tuned routines squeezing the hardware. Layer two: the ecosystem — every framework (PyTorch first), every research repo, every tutorial and Stack Overflow answer assumes CUDA; a generation of engineers thinks in it. Layer three: trust — when a $500M training run hits a bug at 3 a.m., 'it works on NVIDIA' is worth real margin. The moat is switching cost times risk aversion at the worst possible moment.
The tunnels being dug
ROCm reached 'good enough that Microsoft, Meta and OpenAI deploy it' — the HIP layer ports CUDA code, and AMD's 2025–26 rack roadmap comes with software parity claims worth testing against MLPerf. Compilers: Triton lets researchers write kernels in Python that target multiple vendors — PyTorch's compiler emits it by default, quietly commoditizing the kernel layer. Vertical stacks: Google's TPU world (XLA/JAX) simply never touches CUDA, proving frontier models (Gemini) need no NVIDIA software at all — inside one company. Serving-first attacks: inference engines (vLLM, SGLang) abstract the hardware behind an API, and tokens-as-a-service (Groq, Cerebras) hides it entirely.
How to tell if it's working
Watch three indicators: MLPerf submissions on non-NVIDIA hardware (presence = software maturity), what fraction of new open-source projects run day-one on ROCm/TPU, and whether big buyers dual-source frontier training (inference already moves). The moat is real; it is also, for the first time since 2016, under serious siege from three directions at once.
