BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment

• URL: http://arxiv.org/abs/2604.07201v1
• Date: Wed, 08 Apr 2026 15:28:21 GMT
• Title: BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment
• Abstract summary: Best vision-language encoder achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming strong text-only retrievers.<n>We present textbfBRIDGE, a two-component system that resolves this mismatch without multimodal encoders.<n>textbfFORGE is a query alignment model trained via reinforcement learning, which distills noisy multimodal queries into compact, retrieval-optimized search strings.<n>textbfLENS is a reasoning-enhanced dense retriever fine-tuned on reasoning-intensive retrieval data to handle the intent-rich queries FORGE produces.
• Score: 5.285385905661152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal retrieval systems struggle to resolve image-text queries against text-only corpora: the best vision-language encoder achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming strong text-only retrievers. We argue the bottleneck is not the retriever but the query -- raw multimodal queries entangle visual descriptions, conversational noise, and retrieval intent in ways that systematically degrade embedding similarity. We present \textbf{BRIDGE}, a two-component system that resolves this mismatch without multimodal encoders. \textbf{FORGE} (\textbf{F}ocused Retrieval Query Generato\textbf{r}) is a query alignment model trained via reinforcement learning, which distills noisy multimodal queries into compact, retrieval-optimized search strings. \textbf{LENS} (\textbf{L}anguage-\textbf{E}nhanced \textbf{N}eural \textbf{S}earch) is a reasoning-enhanced dense retriever fine-tuned on reasoning-intensive retrieval data to handle the intent-rich queries FORGE produces. Evaluated on MM-BRIGHT (2,803 queries, 29 domains), BRIDGE achieves \textbf{29.7} nDCG@10, surpassing all multimodal encoder baselines including Nomic-Vision (27.6). When FORGE is applied as a plug-and-play aligner on top of Nomic-Vision, the combined system reaches \textbf{33.3} nDCG@10 -- exceeding the best text-only retriever (32.2) -- demonstrating that \textit{query alignment} is the key bottleneck in multimodal-to-text retrieval. https://github.com/mm-bright/multimodal-reasoning-retrieval

Related papers

• SAGE: Semantic Attribute Graphs for Multi-Entity Visual Retrieval [7.2391463583497995]
  We propose SAGE, a training-free framework that parses semantic entities from dense document images.<n>SAGE substantially reduces semantic dilution and outperforms patch-level and OCR-based retrieval baselines.
  arXiv  Detail & Related papers  (2026-09-01T06:07:20Z)
• HIVE: Query, Hypothesize, Verify An LLM Framework for Multimodal Reasoning-Intensive Retrieval [5.5504253907902275]
  Multimodal retrieval models fail on reasoning-intensive queries where images must be deeply integrated with text to identify relevant documents.<n>We introduce textbfHIVE (textbfHypothesis-driven textbfVisual textbfEvidence Retrieval), a plug-and-play framework that injects explicit visual-text reasoning into a retriever via LLMs.
  arXiv  Detail & Related papers  (2026-04-08T15:41:42Z)
• MARVEL: Multimodal Adaptive Reasoning-intensiVe Expand-rerank and retrievaL [5.5504253907902275]
  We introduce textbfMARVEL (textbfMultimodal textbfAdaptive textbfReasoning-intensitextbfVe textbfExpand-rerank and retrievatextbfL), a unified pipeline that combines LLM-driven query expansion, textbfMARVEL-Retriever and GPT-4o-based chain-of-thought reranking.
  arXiv  Detail & Related papers  (2026-04-08T13:35:09Z)
• LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval [0.0]
  LITTA is a query-expansion-centric retrieval framework for evidence page retrieval.<n>Given a user query, LITTA generates complementary query variants using a large language model.<n>Candidates from expanded queries are aggregated through reciprocal rank fusion to improve evidence coverage.
  arXiv  Detail & Related papers  (2026-03-10T13:25:39Z)
• MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval [18.53521844184766]
  MM-BRIGHT is the first multimodal benchmark for reasoning-intensive retrieval.<n>Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains.
  arXiv  Detail & Related papers  (2026-01-14T15:31:54Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering [60.062194349648195]
  Document Visual Question Answering (DocVQA) faces dual challenges in processing lengthy multimodal documents.<n>Current document retrieval-augmented generation (DocRAG) methods remain limited by their text-centric approaches.<n>We introduce MMDocRAG, a comprehensive benchmark featuring 4,055 expert-annotated QA pairs with multi-page, cross-modal evidence chains.
  arXiv  Detail & Related papers  (2025-05-22T09:52:57Z)
• MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrieval [26.585985828583304]
  We introduce MIRe, a retrieval framework that achieves modality interaction without fusing textual features during the alignment.<n>Our method allows the textual query to attend to visual embeddings while not feeding text-driven signals back into the visual representations.<n>Our experiments demonstrate that our pre-training strategy significantly enhances the understanding of multimodal queries.
  arXiv  Detail & Related papers  (2024-11-13T04:32:58Z)
• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs [78.5013630951288]
  This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs)<n>We first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks.<n>Our model, MM-Embed, achieves state-of-the-art performance on the multimodal retrieval benchmark M-BEIR.
  arXiv  Detail & Related papers  (2024-11-04T20:06:34Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognition [94.90258603217008]
  The MultiCoNER RNum2 shared task aims to tackle multilingual named entity recognition (NER) in fine-grained and noisy scenarios. Previous top systems in the MultiCoNER RNum1 either incorporate the knowledge bases or gazetteers. We propose a unified retrieval-augmented system (U-RaNER) for fine-grained multilingual NER.
  arXiv  Detail & Related papers  (2023-05-05T16:59:26Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.