HIVE: Query, Hypothesize, Verify An LLM Framework for Multimodal Reasoning-Intensive Retrieval

• URL: http://arxiv.org/abs/2604.07220v1
• Date: Wed, 08 Apr 2026 15:41:42 GMT
• Title: HIVE: Query, Hypothesize, Verify An LLM Framework for Multimodal Reasoning-Intensive Retrieval
• Abstract summary: Multimodal retrieval models fail on reasoning-intensive queries where images must be deeply integrated with text to identify relevant documents.<n>We introduce textbfHIVE (textbfHypothesis-driven textbfVisual textbfEvidence Retrieval), a plug-and-play framework that injects explicit visual-text reasoning into a retriever via LLMs.
• Score: 5.5504253907902275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal retrieval models fail on reasoning-intensive queries where images (diagrams, charts, screenshots) must be deeply integrated with text to identify relevant documents -- the best multimodal model achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming even strong text-only retrievers (32.2). We introduce \textbf{HIVE} (\textbf{H}ypothesis-driven \textbf{I}terative \textbf{V}isual \textbf{E}vidence Retrieval), a plug-and-play framework that injects explicit visual-text reasoning into a retriever via LLMs. HIVE operates in four stages: (1) initial retrieval over the corpus, (2) LLM-based compensatory query synthesis that explicitly articulates visual and logical gaps observed in top-$k$ candidates, (3) secondary retrieval with the refined query, and (4) LLM verification and reranking over the union of candidates. Evaluated on the multimodal-to-text track of MM-BRIGHT (2,803 real-world queries across 29 technical domains), HIVE achieves a new state-of-the-art aggregated nDCG@10 of \textbf{41.7} -- a \textbf{+9.5} point gain over the best text-only model (DiVeR: 32.2) and \textbf{+14.1} over the best multimodal model (Nomic-Vision: 27.6), where our reasoning-enhanced base retriever contributes 33.2 and the HIVE framework adds a further \textbf{+8.5} points -- with particularly strong results in visually demanding domains (Gaming: 68.2, Chemistry: 42.5, Sustainability: 49.4). Compatible with both standard and reasoning-enhanced retrievers, HIVE demonstrates that LLM-mediated visual hypothesis generation and verification can substantially close the multimodal reasoning gap in retrieval. https://github.com/mm-bright/multimodal-reasoning-retrieval

Related papers

• BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment [5.285385905661152]
  Best vision-language encoder achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming strong text-only retrievers.<n>We present textbfBRIDGE, a two-component system that resolves this mismatch without multimodal encoders.<n>textbfFORGE is a query alignment model trained via reinforcement learning, which distills noisy multimodal queries into compact, retrieval-optimized search strings.<n>textbfLENS is a reasoning-enhanced dense retriever fine-tuned on reasoning-intensive retrieval data to handle the intent-rich queries FORGE produces.
  arXiv  Detail & Related papers  (2026-04-08T15:28:21Z)
• MARVEL: Multimodal Adaptive Reasoning-intensiVe Expand-rerank and retrievaL [5.5504253907902275]
  We introduce textbfMARVEL (textbfMultimodal textbfAdaptive textbfReasoning-intensitextbfVe textbfExpand-rerank and retrievatextbfL), a unified pipeline that combines LLM-driven query expansion, textbfMARVEL-Retriever and GPT-4o-based chain-of-thought reranking.
  arXiv  Detail & Related papers  (2026-04-08T13:35:09Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval [18.53521844184766]
  MM-BRIGHT is the first multimodal benchmark for reasoning-intensive retrieval.<n>Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains.
  arXiv  Detail & Related papers  (2026-01-14T15:31:54Z)
• Comparison of Text-Based and Image-Based Retrieval in Multimodal Retrieval Augmented Generation Large Language Model Systems [0.0]
  We present a comparative analysis of two retrieval approaches for multimodal RAG systems.<n>Direct multimodal embedding retrieval significantly outperforms LLM-summary-based approaches.
  arXiv  Detail & Related papers  (2025-11-20T18:56:49Z)
• MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval [87.24221266746686]
  We introduce MRMR, the first expert-level multidisciplinary multimodal retrieval benchmark requiring intensive reasoning.<n>It challenges retrieval systems across diverse areas of expertise, enabling fine-grained model comparison across domains.<n> queries are reasoning-intensive, with images requiring deeper interpretation such as diagnosing microscopic slides.
  arXiv  Detail & Related papers  (2025-10-10T16:14:56Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning [72.57452266982642]
  OCRBench v2 is a large-scale bilingual text-centric benchmark.<n>It covers 31 diverse scenarios, 10,000 human-verified question-answering pairs, and thorough evaluation metrics.<n>We find that most LMMs score below 50 (100 in total) and suffer from five-type limitations.
  arXiv  Detail & Related papers  (2024-12-31T07:32:35Z)
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs [78.5013630951288]
  This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs)<n>We first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks.<n>Our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR.
  arXiv  Detail & Related papers  (2024-11-04T20:06:34Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.