Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

• URL: http://arxiv.org/abs/2608.12209v1
• Date: Wed, 12 Aug 2026 16:03:31 GMT
• Title: Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction
• Abstract summary: Generation-guided training framework reinterprets visual generation as auxiliary supervision for representation learning.<n>Gas improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension.
• Score: 11.257989394325671
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Multimodal Large Language Models (MLLMs) have achieved remarkable progress, visual understanding and generation are typically treated as divergent objectives. Existing unified frameworks often rely on discrete visual tokenization or diffusion objectives whose generative targets differ from the continuous representations consumed by visual understanding models, making direct transfer to enhance existing pretrained MLLMs non-trivial. In this work, we present GAS, a generation-guided training framework that reinterprets visual generation as auxiliary supervision for representation learning. Concretely, GAS adapts Next Embedding Prediction (NEP) as a cross-modal generation paradigm within a decoupled Mixture-of-Transformers (MoT) architecture. By maintaining a shared lower trunk and parallel upper layers, GAS lets generation losses enrich the shared visual pathway with finer spatial precision and stronger visual retention while shielding the upper understanding layers from direct generation gradients. To maximize this synergy, we further construct highly correlated generation tasks that demand deep cognitive grounding rather than generic synthesis alone. Across model scales and training stages, GAS improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension. Crucially, because the auxiliary generation branch is discarded after training, these gains incur zero inference overhead. Extensive controlled comparisons and representation-level analyses further clarify when and why generation-guided training benefits understanding, and demonstrate the feasibility of generation-guided training as a practical route to stronger multimodal understanding.

Related papers

• Semantic Generative Tuning for Unified Multimodal Models [62.18894352635965]
  Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture.<n> prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives.<n>This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs.
  arXiv  Detail & Related papers  (2026-05-18T17:46:46Z)
• Steering Visual Generation in Unified Multimodal Models with Understanding Supervision [42.765106450407814]
  Unified multimodal models are envisioned to bridge the gap between understanding and generation.<n>We propose Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations.
  arXiv  Detail & Related papers  (2026-05-07T07:20:04Z)
• UniG2U-Bench: Do Unified Models Advance Multimodal Understanding? [50.92401586025528]
  Unified multimodal models have recently demonstrated strong generative capabilities, yet whether and when generation improves understanding remains unclear.<n>We introduce UniG2U-Bench, a comprehensive benchmark categorizing generation-to-understanding (G2U) evaluation into 7 regimes and 30 subtasks.
  arXiv  Detail & Related papers  (2026-03-03T18:36:16Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Growing Visual Generative Capacity for Pre-Trained MLLMs [60.826355079902505]
  Bridge is a pure autoregressive unified MLLM that augments pre-trained visual understanding models with generative ability.<n>We propose a semantic-to-pixel discrete representation that integrates compact semantic tokens with fine-grained pixel tokens.
  arXiv  Detail & Related papers  (2025-10-02T00:40:02Z)
• FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities [76.46448367752944]
  multimodal large language models (MLLMs) unify visual understanding and image generation within a single framework.<n>Most existing MLLMs rely on autore (AR) architectures, which impose inherent limitations on future development.<n>We introduce FUDOKI, a unified multimodal model purely based on discrete flow matching.
  arXiv  Detail & Related papers  (2025-05-26T15:46:53Z)
• Harmonizing Visual Representations for Unified Multimodal Understanding and Generation [53.01486796503091]
  We present emphHarmon, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder.<n>Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T20:50:38Z)
• CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning [24.981279071712173]
  We introduce CAFe, a contrastive-autoregressive fine-tuning framework that enhances LVLMs for both representation and generative tasks.<n>Our approach unifies these traditionally separate tasks, achieving state-of-the-art results in both multimodal retrieval and multimodal generative benchmarks.
  arXiv  Detail & Related papers  (2025-03-25T17:57:17Z)
• VARGPT: Unified Understanding and Generation in a Visual Autoregressive Multimodal Large Language Model [38.61292051733335]
  We present VARGPT, a novel multimodal large language model (MLLM) that unifies visual understanding and generation within a single autoregressive framework.<n>VarGPT employs a next-token prediction paradigm for visual understanding and a next-scale prediction paradigm for visual autoregressive generation.<n> Notably, VARGPT naturally supports capabilities in autoregressive visual generation and instruction-to-image synthesis, showcasing its versatility in both visual understanding and generation tasks.
  arXiv  Detail & Related papers  (2025-01-21T17:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.