Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis

• URL: http://arxiv.org/abs/2603.29620v2
• Date: Wed, 01 Apr 2026 09:31:49 GMT
• Title: Unify-Agent: A Unified Multimodal Agent for World-Grounded Image Synthesis
• Abstract summary: We present Unify-Agent, a unified multimodal agent for world-grounded image synthesis.<n>Our work highlights the value of tightly coupling reasoning, searching, and generation for reliable open-world agentic image synthesis.
• Score: 63.90531780363577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified multimodal models provide a natural and promising architecture for understanding diverse and complex real-world knowledge while generating high-quality images. However, they still rely primarily on frozen parametric knowledge, which makes them struggle with real-world image generation involving long-tail and knowledge-intensive concepts. Inspired by the broad success of agents on real-world tasks, we explore agentic modeling to address this limitation. Specifically, we present Unify-Agent, a unified multimodal agent for world-grounded image synthesis, which reframes image generation as an agentic pipeline consisting of prompt understanding, multimodal evidence searching, grounded recaptioning, and final synthesis. To train our model, we construct a tailored multimodal data pipeline and curate 143K high-quality agent trajectories for world-grounded image synthesis, enabling effective supervision over the full agentic generation process. We further introduce FactIP, a benchmark covering 12 categories of culturally significant and long-tail factual concepts that explicitly requires external knowledge grounding. Extensive experiments show that our proposed Unify-Agent substantially improves over its base unified model across diverse benchmarks and real world generation tasks, while approaching the world knowledge capabilities of the strongest closed-source models. As an early exploration of agent-based modeling for world-grounded image synthesis, our work highlights the value of tightly coupling reasoning, searching, and generation for reliable open-world agentic image synthesis.

Related papers

• ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation [57.673228074553414]
  ToolArtist is a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM)<n>We show that ToolArtist orchestrates reasoning, external tool use, and native image generation within one unified policy.<n> Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines.
  arXiv  Detail & Related papers  (2026-08-05T04:26:19Z)
• Wan-Image: Pushing the Boundaries of Generative Visual Intelligence [86.08534008471356]
  Wan-Image is a unified visual generation system engineered to paradigm-shift image generation models.<n>It is powered by large-scale multi-modal data scaling, a systematic fine-grained annotation engine, and curated reinforcement learning data.<n>Ultimately, Wan-Image revolutionizes visual content creation across e-commerce, entertainment, education, and personal productivity.
  arXiv  Detail & Related papers  (2026-04-21T17:58:58Z)
• DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories [52.57197752244638]
  We introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task.<n>Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues.<n>We construct DISBench, a challenging benchmark built on interconnected visual data.
  arXiv  Detail & Related papers  (2026-02-11T12:51:10Z)
• HiCoGen: Hierarchical Compositional Text-to-Image Generation in Diffusion Models via Reinforcement Learning [66.99487505369254]
  HiCoGen is built upon a novel Chain of Synthesis paradigm.<n>It decomposes complex prompts into minimal semantic units.<n>It then synthesizes these units iteratively, where the image generated in each step provides crucial visual context for the next.<n>Experiments show our approach significantly outperforms existing methods in both concept coverage and compositional accuracy.
  arXiv  Detail & Related papers  (2025-11-25T06:24:25Z)
• Open Multimodal Retrieval-Augmented Factual Image Generation [86.34546873830152]
  We introduce ORIG, an agentic open multimodal retrieval-augmented framework for Factual Image Generation (FIG)<n> ORIG iteratively retrieves and filters multimodal evidence from the web and incrementally integrates the refined knowledge into enriched prompts to guide generation.<n>Experiments demonstrate that ORIG substantially improves factual consistency and overall image quality over strong baselines.
  arXiv  Detail & Related papers  (2025-10-26T04:13:31Z)
• Query-Kontext: An Unified Multimodal Model for Image Generation and Editing [53.765351127477224]
  Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I)<n>We introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal kontext'' composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs.<n> Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases.
  arXiv  Detail & Related papers  (2025-09-30T17:59:46Z)
• Generative World Models of Tasks: LLM-Driven Hierarchical Scaffolding for Embodied Agents [0.0]
  We propose an effective world model for decision-making that models the world's physics and its task semantics.<n>A systematic review of 2024 research in low-resource multi-agent soccer reveals a clear trend towards integrating symbolic and hierarchical methods.<n>We formalize this trend into a framework for Hierarchical Task Environments (HTEs), which are essential for bridging the gap between simple, reactive behaviors and sophisticated, strategic team play.
  arXiv  Detail & Related papers  (2025-09-05T01:03:51Z)
• BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset [140.1967962502411]
  We introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features.<n>A sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation offers practical advantages.<n>Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models.
  arXiv  Detail & Related papers  (2025-05-14T17:11:07Z)
• Grounded Answers for Multi-agent Decision-making Problem through Generative World Model [27.263093790379024]
  generative models often produce sketchy and misleading solutions for complex multi-agent decision-making problems. We show a paradigm that integrates a language-guided simulator into the multi-agent reinforcement learning pipeline to enhance the generated answer. In particular, it can generate consistent interaction sequences and explainable reward functions at interaction states, opening the path for training generative models of the future.
  arXiv  Detail & Related papers  (2024-10-03T16:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.