MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG

• URL: http://arxiv.org/abs/2607.07383v1
• Date: Wed, 08 Jul 2026 13:15:39 GMT
• Title: MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG
• Abstract summary: Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions.<n>We propose MMAgent-R$2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism.
• Score: 50.46218163777298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge-based Visual Question Answering (KB-VQA) requires models to retrieve visual entities matching the query image from large-scale encyclopedic knowledge bases and answer related questions. Existing multimodal Retrieval Augmented Generation (mRAG) methods rely on global visual features to match candidate entities, yet when the knowledge base contains numerous visually similar entities, the retriever struggles to distinguish them, populating the candidate set with visually similar but factually mismatched distractors. Since subsequent processing steps such as noise filtering are also confined to this fixed candidate set, errors from failed retrieval inevitably propagate to the final answer. To address these challenges, we propose MMAgent-R$^2$, an agentic mRAG framework that integrates visual reranking and active rejection as its internal verification mechanism. Visual reranking directly compares query and candidate images, capturing discriminative details beyond textual descriptions to precisely identify the target entity among similar candidates; active rejection discards unreliable results and retrieves additional candidates when no confident match is found, moving beyond the fixed candidate pool. We design a composite reward function with step-level verification rewards and achieve joint optimization of external retrieval, internal verification, and answer generation via GRPO training. Experiments on InfoSeek, E-VQA, and MMhops demonstrate that \ours{} achieves state-of-the-art performance, with particularly notable advantages in challenging retrieval scenarios and complex multi-image multi-hop reasoning tasks.

Related papers

• WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents [53.426679066400844]
  We introduce WeAgent-Harness, a multimodal agentic harness that supports native text-vision interaction and runtime recovery.<n>We develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout.<n>We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image.
  arXiv  Detail & Related papers  (2026-08-28T08:28:43Z)
• Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval [17.695551248974215]
  UniME-R1 is an embedder-adviser framework that learns to reason over initially retrieved candidates.<n>UniME-R1 consistently improves retrieval performance over strong baselines.
  arXiv  Detail & Related papers  (2026-08-06T14:04:56Z)
• Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering [18.5913106358874]
  Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge.<n>Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer.<n>We reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure.
  arXiv  Detail & Related papers  (2026-04-08T14:37:35Z)
• Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models [58.46663983451155]
  PixSearch is an end-to-end Segmenting Large Multimodal Model (LMM) that unifies region-level perception and retrieval-augmented reasoning.<n>During encoding, PixSearch emits search> tokens to trigger retrieval, selects query modalities (text, image, or region), and generates pixel-level masks that directly serve as visual queries.<n>On egocentric and entity-centric VQA benchmarks, PixSearch substantially improves factual consistency and generalization.
  arXiv  Detail & Related papers  (2026-01-27T00:46:08Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Knowledge-based Visual Question Answer with Multimodal Processing, Retrieval and Filtering [55.49652734090316]
  A knowledge-based visual question answering (KB-VQA) requires visual language models (VLMs) to integrate visual understanding with external knowledge retrieval.<n>We propose a novel three-stage method, termed Wiki-PRF, including Processing, Retrieval and Filtering stages.<n> Experiments on benchmark datasets (E-VQA and InfoSeek) show significant improvements in answer quality, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-10-16T12:10:00Z)
• MIXRAG : Mixture-of-Experts Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [6.596018318578605]
  Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by incorporating external knowledge sources during inference.<n>Most existing approaches rely on a single retriever to identify relevant subgraphs, which limits their ability to capture diverse aspects of complex queries.<n>We propose MIXRAG, a Mixture-of-Experts Graph-RAG framework that introduces multiple specialized graph retrievers and a dynamic routing controller.
  arXiv  Detail & Related papers  (2025-09-24T02:44:57Z)
• HANRAG: Heuristic Accurate Noise-resistant Retrieval-Augmented Generation for Multi-hop Question Answering [25.01360941843264]
  HANRAG is a novel framework designed to efficiently tackle problems of varying complexity.<n> Driven by a powerful revelator, HANRAG routes queries, decomposes them into sub-queries, and filters noise from retrieved documents.<n>Results demonstrate that our framework obtains superior performance in both single-hop and multi-hop question-answering tasks.
  arXiv  Detail & Related papers  (2025-09-08T06:22:38Z)
• Divide by Question, Conquer by Agent: SPLIT-RAG with Question-Driven Graph Partitioning [62.640169289390535]
  SPLIT-RAG is a multi-agent RAG framework that addresses the limitations with question-driven semantic graph partitioning and collaborative subgraph retrieval.<n>The innovative framework first create Semantic Partitioning of Linked Information, then use the Type-Specialized knowledge base to achieve Multi-Agent RAG.<n>The attribute-aware graph segmentation manages to divide knowledge graphs into semantically coherent subgraphs, ensuring subgraphs align with different query types.<n>A hierarchical merging module resolves inconsistencies across subgraph-derived answers through logical verifications.
  arXiv  Detail & Related papers  (2025-05-20T06:44:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.