Hybrid Retriever Evolution for Multimodal Document Reasoning Agents

• URL: http://arxiv.org/abs/2606.29648v1
• Date: Sun, 28 Jun 2026 23:36:34 GMT
• Title: Hybrid Retriever Evolution for Multimodal Document Reasoning Agents
• Abstract summary: We introduce a failure-driven evolution framework in which a meta-agent autonomously discovers how a tool-using task agent should coordinate diverse retrievers.<n>On MMLongBench-Doc and DocBench, the evolved agent achieves gains of up to +19.6 points over the unevolved baseline.
• Score: 7.9215284090523355
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Different retrievers, including lexical, semantic, and multimodal approaches, provide highly complementary strengths for multimodal document understanding, yet most systems combine them through fixed pipelines that cannot adapt to the demands of individual reasoning steps. In this work, we ask whether retrieval orchestration itself can be learned as part of the reasoning process. We introduce a failure-driven evolution framework in which a meta-agent autonomously discovers how a tool-using task agent should coordinate diverse retrievers during multi-step document question answering. The meta-agent analyzes incorrect reasoning trajectories, actively probes the same tool environment to diagnose root causes, and iteratively rewrites the task agent's instructions, turning retrieval from a fixed front-end stage into an adaptive, step-wise reasoning decision. The evolved agent learns when to invoke each retriever, how to combine them, and how to compose evidence across modalities and pages. On MMLongBench-Doc and DocBench, the evolved agent achieves gains of up to +19.6 points over the unevolved baseline and consistently outperforms recent systems including MACT, MDocAgent, and SimpleDoc. Detailed retrieval analyses confirm that these improvements arise from adaptive routing and evidence composition rather than reliance on any hard coded retrieval mode, and evolution dynamics reveal a progressive shift from narrow lexical behavior to rich multi-tool coordination. These findings establish autonomous multi-agent coordination as a promising paradigm for multimodal document reasoning.

Related papers

• MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA [23.25080415779712]
  We propose a Memory-Aware Refinement Agent framework that decouples long-document QA into three specialized agents.<n>Across iterations, the agents rely on a dynamically updated structured memory rather than a full accumulated interaction history.<n>Experiments on MMLongBench-Doc and DocBench show that MARDoc achieves strong results, outperforming same-backbone baselines.
  arXiv  Detail & Related papers  (2026-06-04T06:23:01Z)
• Meta-Agent: From Task Descriptions to Verified Multi-Agent Systems [36.485119937780524]
  We present Meta-Agent, a two-phase framework that automatically constructs and executes specialized multi-agent systems.<n>We introduce a three-level error attribution mechanism that distinguishes local, upstream, and structural failures.<n>Experiments against strong multi-agent baselines and ablation studies demonstrate consistent improvements in task success rate, error recovery, and workflow stability.
  arXiv  Detail & Related papers  (2026-05-24T19:38:54Z)
• Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems [65.90791804095561]
  Multi-agent systems address this through structured collaboration among specialized agents, but tighter coordination amplifies a less explored risk.<n>Existing surveys cover individual agent capabilities, multi-agent collaboration, or agent self-evolution separately, leaving the causal dependencies among them unexamined.<n>This survey provides a unified review organized around four causally linked stages, which we term the LIFE progression: Lay the capability foundation, Integrate agents through collaboration, Find faults through attribution, and Evolve through autonomous self-improvement.
  arXiv  Detail & Related papers  (2026-05-14T14:36:13Z)
• TacoMAS: Test-Time Co-Evolution of Topology and Capability in LLM-based Multi-Agent Systems [55.81570336226014]
  We introduce TacoMAS, a test-time co-evolution framework for dynamic multi-agent systems.<n>TacoMAS formulates MAS inference as a task of online graph adaptation, where nodes represent agents with role-specific capabilities and edges define their communication topology.<n>Experiments on four benchmarks demonstrate that TacoMAS outperforms nearly 20 multi-agent baselines, achieving an average improvement of 13.3% over the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-10T13:52:00Z)
• Experience as a Compass: Multi-agent RAG with Evolving Orchestration and Agent Prompts [22.323342919724492]
  HERA is a hierarchical framework that jointly evolves multi-agent orchestration and role-specific agent prompts.<n> HERA achieves an average improvement of 38.69% over recent baselines.<n>Topological analyses reveal emergent self-organization, where sparse exploration yields compact, high-utility multi-agent networks.
  arXiv  Detail & Related papers  (2026-04-01T13:45:52Z)
• ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering [8.852709980681626]
  We present ORCA: Orchestrated Reasoning with Collaborative Agents for Document Visual Question Answering.<n>Our framework leverages a set of specialized AI agents, each dedicated to a distinct modality, enabling fine-language understanding and collaborative reasoning across diverse document components.<n>Our approach achieves significant improvements over state-of-the-art methods, establishing a new paradigm for collaborative agent systems in vision-grained reasoning.
  arXiv  Detail & Related papers  (2026-03-02T22:21:25Z)
• Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detection [59.04089915447622]
  ForenAgent is an interactive IFD framework that enables MLLMs to autonomously generate, execute, and refine Python-based low-level tools around the detection objective.<n>Inspired by human reasoning, we design a dynamic reasoning loop comprising global perception, local focusing, iterative probing, and holistic adjudication.<n>Experiments show that ForenAgent exhibits emergent tool-use competence and reflective reasoning on challenging IFD tasks.
  arXiv  Detail & Related papers  (2025-12-18T08:38:44Z)
• Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding [49.26132236798123]
  Vision Language Models (VLMs) have gradually become a primary approach in document understanding.<n>We propose SLEUTH, a multi agent framework that orchestrates a retriever and four collaborative agents in a coarse to fine process.<n>The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy.
  arXiv  Detail & Related papers  (2025-11-28T03:09:40Z)
• SIRAG: Towards Stable and Interpretable RAG with A Process-Supervised Multi-Agent Framework [7.37561751991963]
  We propose a process-supervised multi-agent framework to bridge the gap between retriever and generator.<n>The proposed framework is modular and plug-and-play, requiring no modification to the retriever or generator.
  arXiv  Detail & Related papers  (2025-09-17T09:09:28Z)
• MAMM-Refine: A Recipe for Improving Faithfulness in Generation with Multi-Agent Collaboration [63.31211701741323]
  We extend multi-agent multi-model reasoning to generation, specifically to improving faithfulness through refinement.<n>We design intrinsic evaluations for each subtask, with our findings indicating that both multi-agent (multiple instances) and multi-model (diverse LLM types) approaches benefit error detection and critiquing.<n>We consolidate these insights into a final "recipe" called Multi-Agent Multi-Model Refinement (MAMM-Refine), where multi-agent and multi-model collaboration significantly boosts performance.
  arXiv  Detail & Related papers  (2025-03-19T14:46:53Z)
• Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking [58.69615583599489]
  Deliberate Thinking based Retriever (Debater) is a novel approach that enhances document representations by incorporating a step-by-step thinking process.<n>Debater significantly outperforms existing methods across several retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-02-18T15:56:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.