Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
Abstract Overview
Mage-VL is a 4B-parameter streaming vision-language model built around Mage-ViT, a codec-native visual encoder that uses motion vectors and residual information to select dynamic patches instead of uniformly sampling frames. The system processes dense anchor frames and sparse predicted frames, then combines these visual tokens with a Qwen3-4B language decoder and a lightweight cognition gate for proactive response triggering in video streams. The paper also describes a five-stage training curriculum and a large-scale data pipeline spanning image captions, image instructions, video captions, and streaming event supervision. Across the paper, the central aim is to improve real-time multimodal understanding and video efficiency by reducing redundant visual computation while preserving temporal context.
Novelty
The main novelty is making streaming multimodal modeling codec-native at the visual front end: Mage-VL allocates tokens according to codec-derived temporal predictability rather than dense per-frame encoding. The work is also distinctive in pairing this sparse codec representation with a proactive speak/silent gate for event-triggered interaction, and in training the visual encoder from scratch on substantially smaller pretraining data than many image-centric baselines.
Results
Mage-VL-4B is reported to match or stay competitive with Qwen3-VL-4B on many static image tasks while improving on a range of video, temporal grounding, and spatial reasoning benchmarks; for example, it scores 83.1 on NextQA versus 79.8, 64.0 on VideoMME versus 59.7, and 57.4 on TimeLens-QVHighlight versus 34.9. The codec-native design reduces visual tokens by about 75% and the paper reports up to 3.5× wall-clock inference speedup over uniform frame sampling, with Mage-VL tc8 cutting NextQA evaluation time from 1460 s to 415 s while also improving accuracy. In streaming evaluation, Mage-VL reaches 55.54% TimVal on SoccerNet-Caption and 64.00 average on OVO-Bench, indicating competitive response timing and real-time perception quality.
Key Points
- Mage-VL replaces uniform frame sampling with codec-guided sparse patch selection, keeping dense anchor-frame context while focusing computation on motion-salient predicted-frame regions.
- The model is trained through a staged curriculum over image, video, codec-stream, and streaming-event data, including an AI-assisted caption optimization pipeline for data construction.
- Empirically, the paper reports stronger efficiency-accuracy tradeoffs than dense frame baselines, especially for long-video understanding, temporal grounding, and streaming interaction.