Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

• URL: http://arxiv.org/abs/2607.24904v1
• Date: Mon, 27 Jul 2026 17:59:53 GMT
• Title: Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
• Abstract summary: We present Mage-VL, an efficient streaming foundation model for real-time multimodal understanding and interaction.<n>At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions.<n>We establish AI4AI data pipelines encompassing prompt-code joint optimization for captioning and AI-driven performance diagnosis.
• Score: 60.85752511708308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.

Related papers

• MOSS-VL Technical Report [66.58420246189313]
  We present MOSS-VL, an open vision-language model family that treats real-time interaction as a first-class capability.<n>It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention.<n> Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets.
  arXiv  Detail & Related papers  (2026-08-15T05:12:53Z)
• Kwai Keye-VL-2.0 Technical Report [53.82434681649277]
  Keye-VL-2.0 is a multimodal foundation model designed to advance long-video understanding and agentic intelligence.<n>It is the first to adapt DeepSeek Sparse Attention (DSA) to GQA-based multimodal architectures.<n>Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL overcomes the algorithmic dilemma of catastrophic forgetting.
  arXiv  Detail & Related papers  (2026-06-09T09:58:08Z)
• FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging [27.981298261747288]
  FlashVID is a training-free acceleration framework for Video Large Language Models (VLLMs)<n>It selects the most representative tokens for basic video representation, then applies Tree-based Stemporal Tokenging (TSTM) for fine-temporal redundancy.<n>FlashVID can serve as a training-free and plug-and-play module for extending long video frames, which enables a 10x increase in video frame input to Qwen2.5-VL.
  arXiv  Detail & Related papers  (2026-02-08T15:56:46Z)
• MiVID: Multi-Strategic Self-Supervision for Video Frame Interpolation using Diffusion Model [2.9795035162522194]
  This article introduces MiVID, a lightweight, self-supervised, diffusion-based framework for video rendering.<n>Our model eliminates the need for explicit motion estimation by combining a 3D U-Net backbone with transformer-style temporal attention.<n>We show that MiVID achieves optimal results just 50 epochs, competitive with several supervised baselines.
  arXiv  Detail & Related papers  (2025-11-08T14:10:04Z)
• AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Model [40.488271586857884]
  AndesVL is a suite of mobile-side MLLMs with 0.6B to 4B parameters based on Qwen3's LLM and various visual encoders.<n>We introduce a 1+N LoRA architecture alongside a Quantization-Aware LoRA Fine-Tuning framework to facilitate efficient task adaptation and model compression.<n>We achieve a 6.7x peak decoding speedup ratio, up to 30.9% memory reduction, and 1.8 bits-per-weight when deploying AndesVL-4B on MediaTek Dimensity 9500 chips.
  arXiv  Detail & Related papers  (2025-10-13T15:04:38Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models [65.37846460916042]
  We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs. We introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency.
  arXiv  Detail & Related papers  (2024-03-11T14:35:32Z)
• An Empirical Study of Training End-to-End Vision-and-Language Transformers [50.23532518166621]
  We present METER(textbfMultimodal textbfEnd-to-end textbfTransformtextbfER), through which we investigate how to design and pre-train a fully transformer-based VL model. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion (e.g., merged attention vs. co-
  arXiv  Detail & Related papers  (2021-11-03T17:55:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.