Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents

• URL: http://arxiv.org/abs/2605.10832v1
• Date: Mon, 11 May 2026 16:49:36 GMT
• Title: Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents
• Abstract summary: Multimodal deep search requires an agent to solve open-world problems by chaining search, tool use, and visual reasoning over evolving textual and visual context.<n>Existing tool-use harnesses treat images returned by search, browsing, or transformation as transient outputs, so intermediate visual evidence cannot be re-consumed by later tools.<n>We introduce a visual-native agent harness centered on an image bank reference protocol, which registers every tool-returned image as an addressable reference.<n>On-policy Data Evolution (ODE) runs a closed-loop data generator that refines itself across rounds from rollouts of
• Score: 23.436701309802533
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal deep search requires an agent to solve open-world problems by chaining search, tool use, and visual reasoning over evolving textual and visual context. Two bottlenecks limit current systems. First, existing tool-use harnesses treat images returned by search, browsing, or transformation as transient outputs, so intermediate visual evidence cannot be re-consumed by later tools. Second, training data is usually built by fixed curation recipes that cannot track the target agent's evolving capability. To address these challenges, we first introduce a visual-native agent harness centered on an image bank reference protocol, which registers every tool-returned image as an addressable reference and makes intermediate visual evidence reusable by later tools. On top of this harness, On-policy Data Evolution (ODE) runs a closed-loop data generator that refines itself across rounds from rollouts of the policy being trained. This per-round refinement makes each round's data target what the current policy still needs to learn. The same framework supports both diverse supervised fine-tuning data and policy-aware reinforcement learning data curation, covering the full training lifecycle of the target agent. Across 8 multimodal deep search benchmarks, ODE improves the Qwen3-VL-8B agent from 24.9% to 39.0% on average, surpassing Gemini-2.5 Pro in standard agent-workflow setting (37.9%). At 30B, ODE raises the average score from 30.6% to 41.5%. Further analyses validate the effectiveness of image-bank reuse, especially on complex tasks requiring iterative visual refinement, while rollout-feedback evolution yields more grounded SFT traces and better policy-matched RL tasks than static synthesis.

Related papers

• WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing [51.8784267764917]
  We present WeAgent-MMGenEdit, a full-stack recipe including a multimodal harness, a scalable data construction pipeline, a comprehensive benchmark, and post-training methods for the agent policy and image backend.<n>We first introduce WeAgent-Harness, a multimodal runtime with persistent evidence management and dedicated verification and integration tools that organize retrieved multimodal evidence into a dense carrier. Upon this, we develop a scalable pipeline for prompt synthesis and agentic trajectory collection, yielding 23K supervised trajectories and 14.7K RL tasks with three-layer verifiable checklists.
  arXiv  Detail & Related papers  (2026-09-04T14:12:03Z)
• WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents [53.426679066400844]
  We introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery.<n>We develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout.<n>We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image.
  arXiv  Detail & Related papers  (2026-08-28T08:28:43Z)
• ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering [26.55005240500685]
  ProMSA is a progressive multimodal search agent for Knowledge-based Visual Question Answering (KB-VQA)<n>Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval.<n>Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy.
  arXiv  Detail & Related papers  (2026-06-26T11:23:18Z)
• MTA-Agent: An Open Recipe for Multimodal Deep Search Agents [51.180338423927985]
  Multi-hop large language models (MLLMs) have demonstrated strong capabilities in visual understanding, yet they remain limited in complex, multi-step reasoning.<n>We propose a Multi-hop Tool-Augmented Agent for Evidence-based QA Synthesis (MTA-Agent)<n>MTA-Agent automatically selects tools and their parameters to retrieve and validate evidence from both visual and textual sources.
  arXiv  Detail & Related papers  (2026-04-07T19:01:45Z)
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents [23.583947864141162]
  EigenData is a hierarchical multi-agent engine that synthesizes tool-grounded dialogues together with executable per-instance checkers.<n>Building on the synthetic data, we develop an RL recipe that first fine-tunes the user model and then applies GRPO-style training.<n>Our results suggest a scalable pathway for bootstrapping complex tool-using behaviors without expensive human annotation.
  arXiv  Detail & Related papers  (2026-01-30T06:01:23Z)
• Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents [52.30603055218294]
  Trajectory2Task is a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios.<n>It converts valid tool-call trajectories into user-facing tasks with controlled intent adaptations.<n>We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures.
  arXiv  Detail & Related papers  (2026-01-28T00:36:13Z)
• Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning [29.605396813225386]
  We show how reinforcement learning can be used to train agents for multi-turn interactive tasks.<n>Our methodology offers a practical approach for training capable agents for multi-turn interactive tasks using open-weight models.
  arXiv  Detail & Related papers  (2025-08-05T14:30:47Z)
• DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors [13.700885996266457]
  Learning from previously collected data via behavioral cloning or offline reinforcement learning (RL) is a powerful recipe for scaling generalist agents. We present theDeepMind Control Visual Benchmark (DMC-VB), a dataset collected in the DeepMind Control Suite to evaluate the robustness of offline RL agents. Accompanying our dataset, we propose three benchmarks to evaluate representation learning methods for pretraining, and carry out experiments on several recently proposed methods.
  arXiv  Detail & Related papers  (2024-09-26T23:07:01Z)
• Retrieval-Augmented Reinforcement Learning [63.32076191982944]
  We train a network to map a dataset of past experiences to optimal behavior. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context. We show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores.
  arXiv  Detail & Related papers  (2022-02-17T02:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.