Uncovering Understanding-Generation Synergy in Native Unified Multimodal Models: From Representation, Task to System
Abstract Overview
This paper studies whether unified multimodal models actually gain from jointly learning visual understanding and image generation, rather than merely exposing both functions through one interface. Using a native pixel-in, pixel-out setting without pretrained vision encoders or VAEs, the authors analyze the interaction at three levels: representation, task, and system. They compare dense sharing, modality-decoupled, and task-decoupled routing designs to isolate when joint learning creates synergy and when it causes interference. Across the study, they argue that synergy is conditional: it depends on architectural specialization, shared task knowledge, and end-to-end optimization.
Novelty
The paper's main novelty is a controlled three-level analysis of understanding-generation synergy in native unified multimodal models, rather than introducing unification only as a product capability. It also proposes a task-decoupled MoT design that separates conflicting visual computation for understanding and generation while preserving semantic interaction between them.
Results
At the representation level, the authors find that generation can enrich features useful for understanding, while understanding can improve vision-language alignment useful for generation, but naive sharing creates asymmetric degradation. Their task-decoupled MoT avoids this trade-off and gives the strongest balanced performance in Table 1. At the task and system levels, joint training improves geometry reasoning, SVG bidirectional mapping, and 3D spatial intelligence, and an end-to-end UMM outperforms a matched planner-executor pipeline on reasoning-intensive image editing benchmarks.
Key Points
- Joint understanding-generation training is not automatically beneficial; its effect depends strongly on routing and parameter-sharing architecture.
- A task-decoupled native UMM can retain semantic interaction while reducing interference between understanding and generation objectives.
- When understanding and generation share underlying knowledge, joint training improves both task performance and end-to-end multimodal systems.
References
- arXiv: https://arxiv.org/abs/2609.01607v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2609.01607v1
- Hugging Face Papers: https://huggingface.co/papers/2609.01607