WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
- URL: http://arxiv.org/abs/2608.28062v2
- Date: Mon, 31 Aug 2026 01:30:21 GMT
- Title: WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents
- Abstract summary: We introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery.<n>We develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout.<n>We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image.
- Score: 53.426679066400844
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search environments often expose retrieved evidence only as text and omit tool-returned images from subsequent context, reducing visually grounded trajectories to text-only reasoning. Long-horizon interaction also compounds tool-call, response-length, timeout, and budget failures, which can discard salvageable trajectories, waste rollout computation, and disturb policy updates. To address these issues, we introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery. Retrieved images receive persistent disk references, allowing the model to inspect, process, and cite them throughout the trajectory. Based on this harness, we develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout. For data construction, a strong MLLM uses WeAgent-Harness to discover, synthesize, and verify MMSearch-style tasks and collect expert trajectories. During post-training, our Failure-Aware GSPO (FA-GSPO) recovers salvageable abnormal rollouts and filters invalid ones to improve bounded multimodal planning and search. We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image, distinguishing image-retrieval failures from visual-perception failures. Evaluation on VisTarget-Bench and seven public benchmarks shows that agentic post-training improves the average score by 19.22 points, enabling our model to outperform similarly sized open-source models and rival models with roughly ten times its parameter count.
Related papers
- WeAgent-MMGenEdit: A Full-Stack Recipe for Multimodal Agentic Image Generation and Editing [51.8784267764917]
We present WeAgent-MMGenEdit, a full-stack recipe including a multimodal harness, a scalable data construction pipeline, a comprehensive benchmark, and post-training methods for the agent policy and image backend.<n>We first introduce WeAgent-Harness, a multimodal runtime with persistent evidence management and dedicated verification and integration tools that organize retrieved multimodal evidence into a dense carrier. Upon this, we develop a scalable pipeline for prompt synthesis and agentic trajectory collection, yielding 23K supervised trajectories and 14.7K RL tasks with three-layer verifiable checklists.
arXiv Detail & Related papers (2026-09-04T14:12:03Z) - SearchOS-V1: Towards Robust Open-Domain Information-Seeking Agent Collaboration [77.28003194268977]
SearchOS is a system-level multi-agent framework that turns fragile, implicit search progress into explicit, persistent, and shared state.<n>Built on SOCM, SearchOS applies a pipeline-parallel scheduling mechanism that overlaps the execution of sub-agents.<n>SearchOS leads all metrics among the evaluated single- and multi-agent baselines.
arXiv Detail & Related papers (2026-07-16T17:51:23Z) - Towards Long-horizon Agentic Multimodal Search [109.0092257657625]
We propose a novel Long-horizon MultiModal deep search framework, named LMM-Searcher, centered on a file-based visual representation mechanism.<n>We equip the agent with a tailored fetch-image tool, enabling a progressive, on-demand visual loading strategy for active perception.<n>Our method successfully scales to 100-turn search horizons, achieving state-of-the-art performance among open-source models.
arXiv Detail & Related papers (2026-04-14T15:40:28Z) - DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories [52.57197752244638]
We introduce DeepImageSearch, a novel agentic paradigm that reformulates image retrieval as an autonomous exploration task.<n>Models must plan and perform multi-step reasoning over raw visual histories to locate targets based on implicit contextual cues.<n>We construct DISBench, a challenging benchmark built on interconnected visual data.
arXiv Detail & Related papers (2026-02-11T12:51:10Z) - ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation [49.01601313084479]
ImAgent is a training-free unified multimodal agent that integrates reasoning, generation, and self-evaluation.<n>Experiments on image generation and editing tasks demonstrate that ImAgent consistently improves over the backbone.
arXiv Detail & Related papers (2025-11-14T17:00:29Z) - MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents [44.63565009665076]
We introduce MMSearch-Plus, a 311-task benchmark that enforces multimodal understanding.<n>We provide a model-agnostic agent framework with standard browsing tools and a set-of-mark (SoM) module.<n>SoM enables provenance-aware zoom-and-retrieve and improves robustness in multi-step reasoning.
arXiv Detail & Related papers (2025-08-29T09:58:27Z) - Towards Text-Image Interleaved Retrieval [49.96332254241075]
We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
arXiv Detail & Related papers (2025-02-18T12:00:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.