Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models

• URL: http://arxiv.org/abs/2405.00175v1
• Date: Tue, 30 Apr 2024 19:51:37 GMT
• Title: Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models
• Authors: Alireza Salemi, Hamed Zamani,
• Abstract summary: uRAG is a framework with a unified retrieval engine that serves multiple downstream retrieval-augmented generation (RAG) systems. We build a large-scale experimentation ecosystem consisting of 18 RAG systems that engage in training and 18 unknown RAG systems that use the uRAG as the new users of the search engine.
• Score: 21.115495457454365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces uRAG--a framework with a unified retrieval engine that serves multiple downstream retrieval-augmented generation (RAG) systems. Each RAG system consumes the retrieval results for a unique purpose, such as open-domain question answering, fact verification, entity linking, and relation extraction. We introduce a generic training guideline that standardizes the communication between the search engine and the downstream RAG systems that engage in optimizing the retrieval model. This lays the groundwork for us to build a large-scale experimentation ecosystem consisting of 18 RAG systems that engage in training and 18 unknown RAG systems that use the uRAG as the new users of the search engine. Using this experimentation ecosystem, we answer a number of fundamental research questions that improve our understanding of promises and challenges in developing search engines for machines.

Related papers

• TURA: Tool-Augmented Unified Retrieval Agent for AI Search [18.427511565701394]
  Traditional RAG approaches struggle with real-time needs and structured queries.<n>We introduce TURA, a novel three-stage framework that combines RAG with agentic tool-use to access both static content and dynamic, real-time information.
  arXiv  Detail & Related papers  (2025-08-06T16:24:17Z)
• Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs [69.10441885629787]
  Retrieval-Augmented Generation (RAG) lifts the factuality of Large Language Models (LLMs) by injecting external knowledge.<n>It falls short on problems that demand multi-step inference; conversely, purely reasoning-oriented approaches often hallucinate or mis-ground facts.<n>This survey synthesizes both strands under a unified reasoning-retrieval perspective.
  arXiv  Detail & Related papers  (2025-07-13T03:29:41Z)
• MMSearch-R1: Incentivizing LMMs to Search [49.889749277236376]
  We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables on-demand, multi-turn search in real-world Internet environments.<n>Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty.
  arXiv  Detail & Related papers  (2025-06-25T17:59:42Z)
• ImpRAG: Retrieval-Augmented Generation with Implicit Queries [49.510101132093396]
  ImpRAG is a query-free RAG system that integrates retrieval and generation into a unified model.<n>We show that ImpRAG achieves 3.6-11.5 improvements in exact match scores on unseen tasks with diverse formats.
  arXiv  Detail & Related papers  (2025-06-02T21:38:21Z)
• Knowing You Don't Know: Learning When to Continue Search in Multi-round RAG through Self-Practicing [4.874077691069634]
  Retrieval Augmented Generation (RAG) has shown strong capability in enhancing language models' knowledge and reducing AI generative hallucinations.<n>Current multi-round RAG systems may continue searching even when enough information has already been retrieved.<n>This paper introduces a new framework, SIM-RAG, to explicitly enhance RAG systems' self-awareness and multi-round retrieval capabilities.
  arXiv  Detail & Related papers  (2025-05-05T17:39:35Z)
• Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines [17.803396998387665]
  Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task. Our model functions both as a generative retriever and an accurate answer generator.
  arXiv  Detail & Related papers  (2025-02-23T16:39:39Z)
• Chain-of-Retrieval Augmented Generation [72.06205327186069]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer. Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• Unanswerability Evaluation for Retrieval Augmented Generation [74.3022365715597]
  UAEval4RAG is a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively. We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries.
  arXiv  Detail & Related papers  (2024-12-16T19:11:55Z)
• Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models [31.769428095250912]
  Auto-RAG is an autonomous iterative retrieval model centered on the reasoning capabilities of Large Language Models (LLMs) We develop a method for autonomously synthesizing reasoning-based decision-making instructions in iterative retrieval. Auto-RAG expresses the iterative retrieval process in natural language, enhancing interpretability.
  arXiv  Detail & Related papers  (2024-11-29T03:01:05Z)
• Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage [74.70255719194819]
  We introduce a novel framework based on sub-question coverage, which measures how well a RAG system addresses different facets of a question. We use this framework to evaluate three commercial generative answer engines: You.com, Perplexity AI, and Bing Chat. We find that while all answer engines cover core sub-questions more often than background or follow-up ones, they still miss around 50% of core sub-questions.
  arXiv  Detail & Related papers  (2024-10-20T22:59:34Z)
• Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation [65.23793829741014]
  Embodied-RAG is a framework that enhances the model of an embodied agent with a non-parametric memory system. At its core, Embodied-RAG's memory is structured as a semantic forest, storing language descriptions at varying levels of detail. We demonstrate that Embodied-RAG effectively bridges RAG to the robotics domain, successfully handling over 200 explanation and navigation queries.
  arXiv  Detail & Related papers  (2024-09-26T21:44:11Z)
• Towards Fair RAG: On the Impact of Fair Ranking in Retrieval-Augmented Generation [53.285436927963865]
  This paper presents the first systematic evaluation of RAG systems integrated with fair rankings. We focus specifically on measuring the fair exposure of each relevant item across the rankings utilized by RAG systems. Our findings indicate that RAG systems with fair rankings can maintain a high level of generation quality and, in many cases, even outperform traditional RAG systems.
  arXiv  Detail & Related papers  (2024-09-17T23:10:04Z)
• A Knowledge-Centric Benchmarking Framework and Empirical Study for Retrieval-Augmented Generation [4.359511178431438]
  Retrieval-Augmented Generation (RAG) enhances generative models by integrating retrieval mechanisms. Despite its advantages, RAG encounters significant challenges, particularly in effectively handling real-world queries. This paper proposes a novel RAG benchmark designed to address these challenges.
  arXiv  Detail & Related papers  (2024-09-03T03:31:37Z)
• Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track [51.25144287084172]
  It is crucial to have an arena to build, test, visualize, and systematically evaluate RAG-based search systems. We propose the TREC 2024 RAG Track to foster innovation in evaluating RAG systems.
  arXiv  Detail & Related papers  (2024-06-24T17:37:52Z)
• FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research [70.6584488911715]
  retrieval-augmented generation (RAG) has attracted considerable research attention. Existing RAG toolkits are often heavy and inflexibly, failing to meet the customization needs of researchers. Our toolkit has implemented 16 advanced RAG methods and gathered and organized 38 benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-22T12:12:40Z)
• The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG) [23.849336345191556]
  The challenge builds upon the MobileCS2 dataset, a real-life customer service datasets with nearly 3000 high-quality dialogs. We define two tasks, track 1 for knowledge retrieval and track 2 for response generation, which are core research questions in dialog systems with RAG. We build baseline systems for the two tracks and design metrics to measure whether the systems can perform accurate retrieval and generate informative and coherent response.
  arXiv  Detail & Related papers  (2024-05-21T07:35:21Z)
• Comparative Analysis of Retrieval Systems in the Real World [0.0]
  The objective is to evaluate and compare various state-of-the-art methods based on their performance in terms of accuracy and efficiency. The analysis explores different combinations of technologies, including Azure Cognitive Search Retriever with GPT-4, Pinecone's Canopy framework, Langchain with Pinecone and different language models. The motivation for this analysis arises from the increasing demand for robust and responsive question-answering systems in various domains.
  arXiv  Detail & Related papers  (2024-05-03T12:30:01Z)
• FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation [31.371489527686578]
  Federated search systems aggregate results from multiple search engines, selecting appropriate sources to enhance result quality and align with user intent. FEB4RAG is a novel dataset specifically designed for federated search within RAG frameworks.
  arXiv  Detail & Related papers  (2024-02-19T07:06:52Z)
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
  Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
  arXiv  Detail & Related papers  (2024-01-30T14:25:32Z)
• Seven Failure Points When Engineering a Retrieval Augmented Generation System [1.8776685617612472]
  RAG systems aim to reduce the problem of hallucinated responses from large language models. RAG systems suffer from limitations inherent to information retrieval systems. We present an experience report on the failure points of RAG systems from three case studies.
  arXiv  Detail & Related papers  (2024-01-11T12:04:11Z)
• Large Language Models for Information Retrieval: A Survey [58.30439850203101]
  Information retrieval has evolved from term-based methods to its integration with advanced neural models. Recent research has sought to leverage large language models (LLMs) to improve IR systems. We delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers.
  arXiv  Detail & Related papers  (2023-08-14T12:47:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.