Steering Visual Generation in Unified Multimodal Models with Understanding Supervision

• URL: http://arxiv.org/abs/2605.05781v1
• Date: Thu, 07 May 2026 07:20:04 GMT
• Title: Steering Visual Generation in Unified Multimodal Models with Understanding Supervision
• Abstract summary: Unified multimodal models are envisioned to bridge the gap between understanding and generation.<n>We propose Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations.
• Score: 42.765106450407814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.

Related papers

• Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System [90.64481646805855]
  unified multimodal models (UMMs) jointly perform visual understanding and generation within a single model.<n>We investigate their relationship at the representation, task, and system levels in a controlled, structurally native setting.<n>We show that the value of UMMs extends beyond a unified interface: appropriate specialization, shared task knowledge, and end-to-end optimization can turn coexistence into synergy.
  arXiv  Detail & Related papers  (2026-09-01T17:59:59Z)
• Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction [11.257989394325671]
  Generation-guided training framework reinterprets visual generation as auxiliary supervision for representation learning.<n>Gas improves aggregate multimodal understanding, with its most reliable gains on perception and spatial comprehension.
  arXiv  Detail & Related papers  (2026-08-12T16:03:31Z)
• Semantic Generative Tuning for Unified Multimodal Models [62.18894352635965]
  Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture.<n> prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives.<n>This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs.
  arXiv  Detail & Related papers  (2026-05-18T17:46:46Z)
• Free Lunch for Unified Multimodal Models: Enhancing Generation via Reflective Rectification with Inherent Understanding [20.397510070808238]
  Unified Multimodal Models (UMMs) aim to integrate visual understanding and generation within a single structure.<n>UMMs exhibit a notable capability mismatch, where their understanding capability significantly outperforms their generation.<n>We propose UniRect-CoT, a training-free unified rectification chain-of-thought framework.
  arXiv  Detail & Related papers  (2026-04-15T06:41:56Z)
• InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editing [87.5130783098133]
  InternVL-U is a lightweight 4B- parameter UMM that democratizes multimodal understanding and reasoning capabilities.<n>It integrates a state-of-the-art Multimodal Large Language Model (MLLM) with a specialized MMDiT-based visual generation head.<n>It consistently outperforms unified baseline models with over 3x larger scales such as BAGEL (14B) on various generation and editing tasks.
  arXiv  Detail & Related papers  (2026-03-10T16:38:33Z)
• Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking [154.2388970262703]
  Unified Vision-Language Models (UVLMs) aim to advance multimodal learning by supporting both understanding and generation within a single framework.<n>We introduce the interleaved Analyzing-Drafting problem-solving loop (AD-Loop), a new think paradigm that alternates between analytic and drafting operations.<n>By interleaving textual thoughts with visual thoughts, AD-Loop enables models to iteratively refine both comprehension and outputs, fostering genuine synergy.
  arXiv  Detail & Related papers  (2026-02-24T23:26:09Z)
• Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation [53.18286807225952]
  Unified Multimodal Models (UMMs) integrate both visual understanding and generation within a single framework.<n>We propose UniMRG (Unified Multi-Representation Generation), a simple yet effective architecture-agnostic post-training method.<n>Our method notably enhances fine-grained perception, reduces hallucinations, and improves spatial understanding, while simultaneously boosting generation capabilities.
  arXiv  Detail & Related papers  (2026-01-29T08:42:25Z)
• Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Models [23.128973540926552]
  Endogenous Reprompting transforms the model's understanding into an explicit generative reasoning step.<n>We show that SEER consistently outperforms state-of-the-art baselines in evaluation accuracy, reprompting efficiency, and generation quality.
  arXiv  Detail & Related papers  (2026-01-28T06:54:36Z)
• Enhancing Semi-Supervised Multi-View Graph Convolutional Networks via Supervised Contrastive Learning and Self-Training [9.300953069946969]
  graph convolutional network (GCN)-based multi-view learning provides a powerful framework for integrating structural information from heterogeneous views.<n>Existing methods often fail to fully exploit the complementary information across views, leading to suboptimal feature representations and limited performance.<n>We propose MV-SupGCN, a semi-supervised GCN model that integrates several complementary components with clear motivations and mutual reinforcement.
  arXiv  Detail & Related papers  (2025-12-15T16:39:23Z)
• RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark [71.3555284685426]
  We introduce RealUnify, a benchmark designed to evaluate bidirectional capability synergy.<n>RealUnify comprises 1,000 meticulously human-annotated instances spanning 10 categories and 32 subtasks.<n>We find that current unified models still struggle to achieve effective synergy, indicating that architectural unification alone is insufficient.
  arXiv  Detail & Related papers  (2025-09-29T15:07:28Z)
• Structure-Regularized Attention for Deformable Object Representation [17.120035855774344]
  Capturing contextual dependencies has proven useful to improve the representational power of deep neural networks. Recent approaches that focus on modeling global context, such as self-attention and non-local operation, achieve this goal by enabling unconstrained pairwise interactions between elements. We consider learning representations for deformable objects which can benefit from context exploitation by modeling the structural dependencies that the data intrinsically possesses.
  arXiv  Detail & Related papers  (2021-06-12T03:10:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.