ReasonIR: Training Retrievers for Reasoning Tasks

• URL: http://arxiv.org/abs/2504.20595v1
• Date: Tue, 29 Apr 2025 09:49:28 GMT
• Title: ReasonIR: Training Retrievers for Reasoning Tasks
• Authors: Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, Luke Zettlemoyer,
• Abstract summary: ReasonIR-8B is the first retriever specifically trained for general reasoning tasks.<n>It achieves a new state-of-the-art of 29.9 nDCG@10 without reranker and 36.9 nDCG@10 with reranker on BRIGHT, a widely-used information retrieval benchmark.
• Score: 139.54343970560103
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present ReasonIR-8B, the first retriever specifically trained for general reasoning tasks. Existing retrievers have shown limited gains on reasoning tasks, in part because existing training datasets focus on short factual queries tied to documents that straightforwardly answer them. We develop a synthetic data generation pipeline that, for each document, our pipeline creates a challenging and relevant query, along with a plausibly related but ultimately unhelpful hard negative. By training on a mixture of our synthetic data and existing public data, ReasonIR-8B achieves a new state-of-the-art of 29.9 nDCG@10 without reranker and 36.9 nDCG@10 with reranker on BRIGHT, a widely-used reasoning-intensive information retrieval (IR) benchmark. When applied to RAG tasks, ReasonIR-8B improves MMLU and GPQA performance by 6.4% and 22.6% respectively, relative to the closed-book baseline, outperforming other retrievers and search engines. In addition, ReasonIR-8B uses test-time compute more effectively: on BRIGHT, its performance consistently increases with longer and more information-rich rewritten queries; it continues to outperform other retrievers when combined with an LLM reranker. Our training recipe is general and can be easily extended to future LLMs; to this end, we open-source our code, data, and model.

Related papers

• Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance [24.842839260409075]
  In this work we forgo real training documents and annotations altogether.<n>We use open-source LLMs to directly generate synthetic documents that answer real user queries according to several different levels of relevance.<n> Experiments on various IR datasets show that our proposed approach outperforms conventional training with InfoNCE by a large margin.
  arXiv  Detail & Related papers  (2025-03-29T22:33:22Z)
• Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning [76.50690734636477]
  We introduce Rank-R1, a novel LLM-based reranker that performs reasoning over both the user query and candidate documents before performing the ranking task.<n>Our experiments on the TREC DL and BRIGHT datasets show that Rank-R1 is highly effective, especially for complex queries.
  arXiv  Detail & Related papers  (2025-03-08T03:14:26Z)
• Can we Retrieve Everything All at Once? ARM: An Alignment-Oriented LLM-based Retrieval Method [48.14236175156835]
  ARM aims to better align the question with the organization of the data collection by exploring relationships among data objects.<n>It outperforms standard RAG with query decomposition by up to 5.2 pt in execution accuracy and agentic RAG (ReAct) by up to 15.9 pt.<n>It achieves up to 5.5 pt and 19.3 pt higher F1 match scores compared to these approaches.
  arXiv  Detail & Related papers  (2025-01-30T18:07:19Z)
• W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answering [28.79851078451609]
  We propose W-RAG, a method that draws weak training signals from the downstream task and fine-tunes the retriever to prioritize passages that most benefit the task.<n>We conduct comprehensive experiments across four publicly available OpenQA datasets to demonstrate that our approach enhances both retrieval and OpenQA performance.
  arXiv  Detail & Related papers  (2024-08-15T22:34:44Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• RAR-b: Reasoning as Retrieval Benchmark [7.275757292756447]
  We transform reasoning tasks into retrieval tasks to evaluate reasoning abilities stored in retriever models. Recent decoder-based embedding models show great promise in narrowing the gap. We release Reasoning as Retrieval Benchmark (RAR-b), a holistic suite of tasks and settings to evaluate the reasoning abilities stored in retriever models.
  arXiv  Detail & Related papers  (2024-04-09T14:34:48Z)
• Saving Dense Retriever from Shortcut Dependency in Conversational Search [7.584170081762014]
  A retrieval shortcut in conversational search (CS) causes models to retrieve passages solely relying on partial history while disregarding the latest question. We show naively trained dense retrievers heavily exploit the shortcut and hence perform poorly when asked to answer history-independent questions. To prevent models from solely relying on the shortcut, we explore iterative hard negatives mined by pre-trained dense retrievers.
  arXiv  Detail & Related papers  (2022-02-15T09:53:35Z)
• Learning to Retrieve Passages without Supervision [58.31911597824848]
  Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs. We investigate whether dense retrievers can be learned in a self-supervised fashion, and applied effectively without any annotations.
  arXiv  Detail & Related papers  (2021-12-14T19:18:08Z)
• Adversarial Retriever-Ranker for dense text retrieval [51.87158529880056]
  We present Adversarial Retriever-Ranker (AR2), which consists of a dual-encoder retriever plus a cross-encoder ranker. AR2 consistently and significantly outperforms existing dense retriever methods. This includes the improvements on Natural Questions R@5 to 77.9%(+2.1%), TriviaQA R@5 to 78.2%(+1.4), and MS-MARCO MRR@10 to 39.5%(+1.3%)
  arXiv  Detail & Related papers  (2021-10-07T16:41:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.