Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training

• URL: http://arxiv.org/abs/2603.25706v2
• Date: Mon, 30 Mar 2026 03:26:27 GMT
• Title: Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training
• Abstract summary: We introduce a framework consisting of a planner and a visualizer.<n>The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly.<n>These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency.
• Score: 68.94182767962914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent unified models have made unprecedented progress in both understanding and generation. However, while most of them accept multi-modal inputs, they typically produce only single-modality outputs. This challenge of producing interleaved content is mainly due to training data scarcity and the difficulty of modeling long-range cross-modal context. To address this issue, we decompose interleaved generation into textual planning and visual consistency modeling, and introduce a framework consisting of a planner and a visualizer. The planner produces dense textual descriptions for visual content, while the visualizer synthesizes images accordingly. Under this guidance, we construct large-scale textual-proxy interleaved data (where visual content is represented in text) to train the planner, and curate reference-guided image data to train the visualizer. These designs give rise to Wan-Weaver, which exhibits emergent interleaved generation ability with long-range textual coherence and visual consistency. Meanwhile, the integration of diverse understanding and generation data into planner training enables Wan-Weaver to achieve robust task reasoning and generation proficiency. To assess the model's capability in interleaved generation, we further construct a benchmark that spans a wide range of use cases across multiple dimensions. Extensive experiments demonstrate that, even without access to any real interleaved data, Wan-Weaver achieves superior performance over existing methods.

Related papers

• FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching [86.31254356971506]
  FlowInOne is a framework that reformulates multimodal generation as a purely visual flow.<n>It unifies text-to-image generation, layout-guided editing, and visual instruction following under one coherent paradigm.<n>It achieves state-of-the-art performance across all unified generation tasks, surpassing both open-source models and competitive commercial systems.
  arXiv  Detail & Related papers  (2026-04-08T07:22:08Z)
• Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization [35.14373974143734]
  We propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models.<n>Our approach jointly models text and image generation within a single decoding trajectory and optimize it with our novel hybrid rewards.<n>Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.
  arXiv  Detail & Related papers  (2026-03-10T11:49:20Z)
• DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning [16.880791276029964]
  "Thinking with images" is a shift in Vision Language Models from text-dominant chain-of-thought to image-interactive reasoning.<n>We present DeepSketcher, a comprehensive suite comprising both an image-text interleaved dataset and a self-contained model.<n>We design a model that performs interleaved image-text reasoning and generates "visual thoughts" by operating directly in the visual embedding space.
  arXiv  Detail & Related papers  (2025-09-30T07:02:01Z)
• Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation [54.588082888166504]
  We present Mogao, a unified framework that enables interleaved multi-modal generation through a causal approach.<n>Mogoo integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance.<n>Experiments show that Mogao achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs.
  arXiv  Detail & Related papers  (2025-05-08T17:58:57Z)
• VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning [68.98988753763666]
  We propose VisualCloze, a universal image generation framework.<n>VisualCloze supports a wide range of in-domain tasks, generalization to unseen ones, unseen unification of multiple tasks, and reverse generation.<n>We introduce Graph200K, a graph-structured dataset that establishes various interrelated tasks, enhancing task density and transferable knowledge.
  arXiv  Detail & Related papers  (2025-04-10T17:59:42Z)
• WeGen: A Unified Model for Interactive Multimodal Generation as We Chat [51.78489661490396]
  We introduce WeGen, a model that unifies multimodal generation and understanding.<n>It can generate diverse results with high creativity for less detailed instructions.<n>We show it achieves state-of-the-art performance across various visual generation benchmarks.
  arXiv  Detail & Related papers  (2025-03-03T02:50:07Z)
• ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer [40.32254040909614]
  We propose ACE, an All-round Creator and Editor, for visual generation tasks. We first introduce a unified condition format termed Long-context Condition Unit (LCU) We then propose a novel Transformer-based diffusion model that uses LCU as input, aiming for joint training across various generation and editing tasks.
  arXiv  Detail & Related papers  (2024-09-30T17:56:27Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.