Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward

• URL: http://arxiv.org/abs/2511.20561v2
• Date: Sat, 29 Nov 2025 16:01:48 GMT
• Title: Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
• Authors: Yuwei Niu, Weiyang Jin, Jiaqi Liao, Chaoran Feng, Peng Jin, Bin Lin, Zongjian Li, Bin Zhu, Weihao Yu, Li Yuan,
• Abstract summary: We introduce UniSandbox, a decoupled evaluation framework paired with controlled, synthetic datasets to avoid data leakage.<n>Our findings reveal a significant understanding-generation gap, which is mainly reflected in two key dimensions: reasoning generation and knowledge transfer.<n>UniSandbox provides preliminary insights for designing future unified architectures and training strategies that truly bridge the gap between understanding and generation.
• Score: 33.56471468540189
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent years have witnessed significant progress in Unified Multimodal Models, yet a fundamental question remains: Does understanding truly inform generation? To investigate this, we introduce UniSandbox, a decoupled evaluation framework paired with controlled, synthetic datasets to avoid data leakage and enable detailed analysis. Our findings reveal a significant understanding-generation gap, which is mainly reflected in two key dimensions: reasoning generation and knowledge transfer. Specifically, for reasoning generation tasks, we observe that explicit Chain-of-Thought (CoT) in the understanding module effectively bridges the gap, and further demonstrate that a self-training approach can successfully internalize this ability, enabling implicit reasoning during generation. Additionally, for knowledge transfer tasks, we find that CoT assists the generative process by helping retrieve newly learned knowledge, and also discover that query-based architectures inherently exhibit latent CoT-like properties that affect this transfer. UniSandbox provides preliminary insights for designing future unified architectures and training strategies that truly bridge the gap between understanding and generation. Code and data are available at https://github.com/PKU-YuanGroup/UniSandBox

Related papers

• Synergizing Understanding and Generation with Interleaved Analyzing-Drafting Thinking [154.2388970262703]
  Unified Vision-Language Models (UVLMs) aim to advance multimodal learning by supporting both understanding and generation within a single framework.<n>We introduce the interleaved Analyzing-Drafting problem-solving loop (AD-Loop), a new think paradigm that alternates between analytic and drafting operations.<n>By interleaving textual thoughts with visual thoughts, AD-Loop enables models to iteratively refine both comprehension and outputs, fostering genuine synergy.
  arXiv  Detail & Related papers  (2026-02-24T23:26:09Z)
• Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Models [23.529904770014735]
  This paper introduces a novel perspective: leveraging understanding to enhance the fidelity and detail richness of generated images.<n>We propose Forge-and-Quench, a new unified framework that puts this principle into practice.<n>Experiments show that Forge-and-Quench significantly improves image fidelity and detail across multiple models.
  arXiv  Detail & Related papers  (2026-01-08T08:18:44Z)
• A Survey on Generative Recommendation: Data, Model, and Tasks [55.36322811257545]
  generative recommendation reconceptualizes recommendation as a generation task rather than discriminative scoring.<n>This survey provides a comprehensive examination through a unified tripartite framework spanning data, model, and task dimensions.<n>We identify five key advantages: world knowledge integration, natural language understanding, reasoning capabilities, scaling laws, and creative generation.
  arXiv  Detail & Related papers  (2025-10-31T04:02:58Z)
• Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generation [18.99847259801634]
  We propose Reinforcement Learning from Augmented Generation (RLAG) to embed domain knowledge into large language models.<n>Our approach iteratively cycles between sampling generations and optimize the model through calculated rewards.<n> Experimental results across medical, legal, astronomy, and current events datasets demonstrate that our proposed method significantly outperforms baseline approaches.
  arXiv  Detail & Related papers  (2025-09-24T14:30:16Z)
• UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation [39.921363034430875]
  Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence.<n>We study the modality alignment behaviors of task-specific expert models for understanding and generation.<n>We introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference.
  arXiv  Detail & Related papers  (2025-06-20T17:52:31Z)
• PropMEND: Hypernetworks for Knowledge Propagation in LLMs [82.99849359892112]
  We present a hypernetwork-based approach for knowledge propagation, named PropMEND.<n>We show almost 2x accuracy on challenging multi-hop questions whose answers are not explicitly stated in the injected fact.<n>We also introduce a new dataset, Controlled RippleEdit, to evaluate the generalization of our hypernetwork.
  arXiv  Detail & Related papers  (2025-06-10T15:44:19Z)
• Are Unified Vision-Language Models Necessary: Generalization Across Understanding and Generation [50.22361866757033]
  unified vision-language models (VLMs) integrate both visual understanding and generation capabilities.<n>This paper systematically investigates the generalization across understanding and generation tasks in unifiedVLMs.
  arXiv  Detail & Related papers  (2025-05-29T03:40:21Z)
• Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation [89.65955788873532]
  Open-domain question answering (OpenQA) represents a cornerstone in natural language processing (NLP)<n>We propose a novel framework named GenKI, which aims to improve the OpenQA performance by exploring Knowledge Integration and controllable Generation.
  arXiv  Detail & Related papers  (2025-05-26T08:18:33Z)
• UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens [54.40871421476035]
  We present UniCTokens, a framework that integrates personalized information into a unified vision language model (VLM) for understanding and generation.<n>UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks.<n>Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding.
  arXiv  Detail & Related papers  (2025-05-20T17:56:01Z)
• DP-CRE: Continual Relation Extraction via Decoupled Contrastive Learning and Memory Structure Preservation [4.303714963263037]
  Continuous Relation Extraction (CRE) aims to incrementally learn relation knowledge from a non-stationary stream of data. Current replay-based training paradigms prioritize all data uniformly and train memory samples through multiple rounds. We introduce the DecouPled CRE framework that decouples the process of prior information preservation and new knowledge acquisition.
  arXiv  Detail & Related papers  (2024-03-05T07:16:51Z)
• KAT: A Knowledge Augmented Transformer for Vision-and-Language [56.716531169609915]
  We propose a novel model - Knowledge Augmented Transformer (KAT) - which achieves a strong state-of-the-art result on the open-domain multimodal task of OK-VQA. Our approach integrates implicit and explicit knowledge in an end to end encoder-decoder architecture, while still jointly reasoning over both knowledge sources during answer generation. An additional benefit of explicit knowledge integration is seen in improved interpretability of model predictions in our analysis.
  arXiv  Detail & Related papers  (2021-12-16T04:37:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.