CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration

• URL: http://arxiv.org/abs/2607.05465v1
• Date: Mon, 06 Jul 2026 04:57:18 GMT
• Title: CanvasAgent: Enabling Complex Image Creation and Editing via Visual Tool Orchestration
• Abstract summary: We introduce CanvasCraft, a large-scale multimodal tool-use dataset for complex image creation and editing.<n>We also present CanvasAgent, a toolaugmented multimodal agent that learns to orchestrate heterogeneous visual tools through multi-turn interaction.
• Score: 11.992882604393003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Complex image creation and editing often require more than a single generation or editing model. A user request may involve synthesizing images, localizing objects, segmenting regions, editing selected content, compositing intermediate assets, reading text, and enhancing the final result. Such tasks shift multimodal agents from perception-augmented reasoning to manipulation-centered visual creation, where tools must actively transform visual states rather than merely inspect them. However, existing multimodal tool-use agents are mostly optimized for perception, search, or domain-specific editing, and lack large-scale supervision for executable image-creation trajectories. In this paper, we introduce CanvasCraft, a large-scale multimodal tool-use dataset for complex image creation and editing, and \textbf{CanvasAgent}, a tool-augmented multimodal agent that learns to orchestrate heterogeneous visual tools through multi-turn interaction. CanvasCraft contains 140K fully annotated executable trajectories and 10K RL task specifications. CanvasAgent is first trained with SFT to learn executable reasoning-action trajectories, and is then optimized with GRPO using a hybrid reward that combines outcome- and process-level signals. During rollout, CanvasAgent inspects intermediate results, tracks visual assets, and adapts tool decisions to the evolving visual state. Experiments evaluate both final image quality and trajectory behavior, demonstrating the effectiveness of CanvasAgent and the proposed dataset for complex multi-tool image creation workflows.

Related papers

• WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing [51.8784267764917]
  We present WeAgent-MMGenEdit, a full-stack recipe including a multimodal harness, a scalable data construction pipeline, a comprehensive benchmark, and post-training methods for the agent policy and image backend.<n>We first introduce WeAgent-Harness, a multimodal runtime with persistent evidence management and dedicated verification and integration tools that organize retrieved multimodal evidence into a dense carrier. Upon this, we develop a scalable pipeline for prompt synthesis and agentic trajectory collection, yielding 23K supervised trajectories and 14.7K RL tasks with three-layer verifiable checklists.
  arXiv  Detail & Related papers  (2026-09-04T14:12:03Z)
• ToolArtist: Tool-Using Unified Multimodal Models for Agentic Image Generation [57.673228074553414]
  ToolArtist is a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM)<n>We show that ToolArtist orchestrates reasoning, external tool use, and native image generation within one unified policy.<n> Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines.
  arXiv  Detail & Related papers  (2026-08-05T04:26:19Z)
• IEA: Amateur-Friendly Conversational Image Editing Agent via Three Stages of Multitask Alignment [16.199788035961564]
  We propose IEA, a conversational Image Editing Agent that learns to operate parameterized tools in an explicit, interpretable action space.<n>IEA is trained via a three-stage multitask pipeline: SFT on distilled expert edits, GRPO with rewards for likeness improvement, tool usefulness, and intent summarization, and (3) large-scale synthetic fine-tuning to jointly master image editing, refinement, and user intent summarization.
  arXiv  Detail & Related papers  (2026-06-06T07:11:08Z)
• coDrawAgents: A Multi-Agent Dialogue Framework for Compositional Image Generation [48.027946344020314]
  We propose coDrawAgents, an interactive multi-agent dialogue framework with four specialized agents.<n>The Interpreter decides between a direct text-to-image pathway and a layout-aware multi-agent process.<n>The Planner proposes layouts for objects with the same semantic priority level while grounding decisions in the evolving visual context.<n>The Checker introduces an explicit error-correction mechanism by validating spatial consistency and attribute alignment.<n>The Painter synthesizes the image step by step, incorporating newly planned objects into the canvas to provide richer context for subsequent iterations.
  arXiv  Detail & Related papers  (2026-03-13T09:32:06Z)
• IMAGAgent: Orchestrating Multi-Turn Image Editing via Constraint-Aware Planning and Reflection [40.21337735524356]
  IMAGAgent is a multi-turn image editing agent framework based on a "plan-execute-reflect" closed-loop mechanism.<n>It achieves deep synergy among instruction parsing, tool scheduling, and adaptive correction within a unified pipeline.<n>Experiments on our constructed textbfMTEditBench and the MagicBrush dataset demonstrate that IMAGAgent achieves performance significantly superior to existing methods.
  arXiv  Detail & Related papers  (2026-02-12T02:37:38Z)
• Training Multi-Image Vision Agents via End2End Reinforcement Learning [51.81337984526068]
  We propose IMAgent, an open-source vision agent trained via end-to-end reinforcement learning.<n>By leveraging a multi-agent system, we generate challenging and visually-rich multi-image QA pairs.<n>We develop two specialized tools for visual reflection and confirmation, allowing the model to proactively reallocate its attention to image content.
  arXiv  Detail & Related papers  (2025-12-05T10:02:38Z)
• PixelCraft: A Multi-Agent System for High-Fidelity Visual Reasoning on Structured Images [58.73779101355669]
  PixelCraft is a novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images.<n>Building on this foundation, PixelCraft facilitates visual reasoning through a dynamic three-stage workflow of tool selection, agent discussion, and self-criticism.
  arXiv  Detail & Related papers  (2025-09-29T17:59:49Z)
• Marmot: Object-Level Self-Correction via Multi-Agent Reasoning [55.74860093731475]
  Marmot is a novel and generalizable framework that leverages Multi-Agent Reasoning for Multi-Object Self-Correcting.<n>Marmot significantly improves accuracy in object counting, attribute assignment, and spatial relationships for image generation tasks.
  arXiv  Detail & Related papers  (2025-04-10T16:54:28Z)
• Group Diffusion Transformers are Unsupervised Multitask Learners [49.288489286276146]
  Group Diffusion Transformers (GDTs) are a novel framework that unifies diverse visual generation tasks. GDTs build upon diffusion transformers with minimal architectural modifications by concatenating self-attention tokens across images. We evaluate GDTs on a benchmark featuring over 200 instructions across 30 distinct visual generation tasks.
  arXiv  Detail & Related papers  (2024-10-19T07:53:15Z)
• GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing [60.09562648953926]
  GenArtist is a unified image generation and editing system coordinated by a multimodal large language model (MLLM) agent. We integrate a comprehensive range of existing models into the tool library and utilize the agent for tool selection and execution. Experiments demonstrate that GenArtist can perform various generation and editing tasks, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-07-08T04:30:53Z)
• Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation [72.6168579583414]
  CompAgent is a training-free approach for compositional text-to-image generation with a large language model (LLM) agent as its core. Our approach achieves more than 10% improvement on T2I-CompBench, a comprehensive benchmark for open-world compositional T2I generation.
  arXiv  Detail & Related papers  (2024-01-28T16:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.