Multimodality
Concept
Core Technical
One model, many senses: text, image, audio, video in a single network.
You're looking at a concept explainer: what this term means in plain English, and the records it ties together.
Modern frontier models are natively multimodal — trained on interleaved text, images and audio so the same weights can watch, listen, read and respond. GPT-4o's real-time voice and Gemini's video understanding mark the shift; robotics VLAs extend it to action. Each added modality multiplies training-data and compute appetite.
How this connects to the rest of the stack
This record's links at a glance — click a group to jump to it.
Sources
Record reviewed as of 2026-01-15 — see Methodology.