Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation

• URL: http://arxiv.org/abs/2606.30054v1
• Date: Mon, 29 Jun 2026 09:45:15 GMT
• Title: Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation
• Abstract summary: We introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation.<n>ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences.
• Score: 65.04930747239537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.

Related papers

• Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization [35.14373974143734]
  We propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models.<n>Our approach jointly models text and image generation within a single decoding trajectory and optimize it with our novel hybrid rewards.<n>Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.
  arXiv  Detail & Related papers  (2026-03-10T11:49:20Z)
• Unified Text-Image Generation with Weakness-Targeted Post-Training [57.956648078400775]
  Unified multimodal generation architectures that jointly produce text and images have recently emerged as a promising direction for text-to-image (T2I) synthesis.<n>This work explores post-training to achieve fully unified text-image generation, where models autonomously transition from textual reasoning to visual synthesis.
  arXiv  Detail & Related papers  (2026-01-07T19:19:44Z)
• Query-Kontext: An Unified Multimodal Model for Image Generation and Editing [53.765351127477224]
  Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I)<n>We introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal kontext'' composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs.<n> Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases.
  arXiv  Detail & Related papers  (2025-09-30T17:59:46Z)
• MENTOR: Efficient Multimodal-Conditioned Tuning for Autoregressive Vision Generation Models [30.494968865008513]
  Recent text-to-image models struggle with precise visual control, balancing multimodal inputs, and requiring extensive training for complex image generation.<n>We propose MENTOR, a novel framework for efficient Multimodal-conditioned Tuning for Autoregressive multimodal image generation.<n>Our method delivers superior image reconstruction fidelity, broad task adaptability, and improved training efficiency compared to diffusion-based methods.
  arXiv  Detail & Related papers  (2025-07-13T10:52:59Z)
• BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset [140.1967962502411]
  We introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features.<n>A sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation offers practical advantages.<n>Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models.
  arXiv  Detail & Related papers  (2025-05-14T17:11:07Z)
• Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation [54.588082888166504]
  We present Mogao, a unified framework that enables interleaved multi-modal generation through a causal approach.<n>Mogoo integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance.<n>Experiments show that Mogao achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs.
  arXiv  Detail & Related papers  (2025-05-08T17:58:57Z)
• Unified Discrete Diffusion for Simultaneous Vision-Language Generation [78.21352271140472]
  We present a unified multimodal generation model that can conduct both the "modality translation" and "multi-modality generation" tasks. Specifically, we unify the discrete diffusion process for multimodal signals by proposing a unified transition matrix. Our proposed method can perform comparably to the state-of-the-art solutions in various generation tasks.
  arXiv  Detail & Related papers  (2022-11-27T14:46:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.