Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models
- URL: http://arxiv.org/abs/2405.00175v1
- Date: Tue, 30 Apr 2024 19:51:37 GMT
- Title: Towards a Search Engine for Machines: Unified Ranking for Multiple Retrieval-Augmented Large Language Models
- Authors: Alireza Salemi, Hamed Zamani,
- Abstract summary: uRAG is a framework with a unified retrieval engine that serves multiple downstream retrieval-augmented generation (RAG) systems.
We build a large-scale experimentation ecosystem consisting of 18 RAG systems that engage in training and 18 unknown RAG systems that use the uRAG as the new users of the search engine.
- Score: 21.115495457454365
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper introduces uRAG--a framework with a unified retrieval engine that serves multiple downstream retrieval-augmented generation (RAG) systems. Each RAG system consumes the retrieval results for a unique purpose, such as open-domain question answering, fact verification, entity linking, and relation extraction. We introduce a generic training guideline that standardizes the communication between the search engine and the downstream RAG systems that engage in optimizing the retrieval model. This lays the groundwork for us to build a large-scale experimentation ecosystem consisting of 18 RAG systems that engage in training and 18 unknown RAG systems that use the uRAG as the new users of the search engine. Using this experimentation ecosystem, we answer a number of fundamental research questions that improve our understanding of promises and challenges in developing search engines for machines.
Related papers
- Enhancing Retrieval and Managing Retrieval: A Four-Module Synergy for Improved Quality and Efficiency in RAG Systems [14.62114319247837]
Retrieval-augmented generation (RAG) techniques leverage the in-context learning capabilities of large language models (LLMs) to produce more accurate and relevant responses.
A critical component, the Query Rewriter module, enhances knowledge retrieval by generating a search-friendly query.
These four RAG modules synergistically improve the response quality and efficiency of the RAG system.
arXiv Detail & Related papers (2024-07-15T12:35:00Z) - Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track [51.25144287084172]
It is crucial to have an arena to build, test, visualize, and systematically evaluate RAG-based search systems.
We propose the TREC 2024 RAG Track to foster innovation in evaluating RAG systems.
arXiv Detail & Related papers (2024-06-24T17:37:52Z) - Comparative Analysis of Retrieval Systems in the Real World [0.0]
The objective is to evaluate and compare various state-of-the-art methods based on their performance in terms of accuracy and efficiency.
The analysis explores different combinations of technologies, including Azure Cognitive Search Retriever with GPT-4, Pinecone's Canopy framework, Langchain with Pinecone and different language models.
The motivation for this analysis arises from the increasing demand for robust and responsive question-answering systems in various domains.
arXiv Detail & Related papers (2024-05-03T12:30:01Z) - RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems [51.171355532527365]
We introduce the RAGGED framework to analyze and optimize RAG systems.
We study two classic sparse and dense retrievers, and four top-performing LMs in encoder-decoder and decoder-only architectures.
arXiv Detail & Related papers (2024-03-14T02:26:31Z) - FeB4RAG: Evaluating Federated Search in the Context of Retrieval
Augmented Generation [31.371489527686578]
Federated search systems aggregate results from multiple search engines, selecting appropriate sources to enhance result quality and align with user intent.
FEB4RAG is a novel dataset specifically designed for federated search within RAG frameworks.
arXiv Detail & Related papers (2024-02-19T07:06:52Z) - Distillation Enhanced Generative Retrieval [96.69326099136289]
Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target.
In this work, we identify a viable direction to further enhance generative retrieval via distillation and propose a feasible framework, named DGR.
We conduct experiments on four public datasets, and the results indicate that DGR achieves state-of-the-art performance among the generative retrieval methods.
arXiv Detail & Related papers (2024-02-16T15:48:24Z) - CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources.
This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
arXiv Detail & Related papers (2024-01-30T14:25:32Z) - Seven Failure Points When Engineering a Retrieval Augmented Generation
System [1.8776685617612472]
RAG systems aim to reduce the problem of hallucinated responses from large language models.
RAG systems suffer from limitations inherent to information retrieval systems.
We present an experience report on the failure points of RAG systems from three case studies.
arXiv Detail & Related papers (2024-01-11T12:04:11Z) - Large Language Models for Information Retrieval: A Survey [57.7992728506871]
Information retrieval has evolved from term-based methods to its integration with advanced neural models.
Recent research has sought to leverage large language models (LLMs) to improve IR systems.
We delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers.
arXiv Detail & Related papers (2023-08-14T12:47:22Z) - Multi-Grained Knowledge Retrieval for End-to-End Task-Oriented Dialog [42.088274728084265]
Retrieving proper domain knowledge from an external database lies at the heart of end-to-end task-oriented dialog systems.
Most existing systems blend knowledge retrieval with response generation and optimize them with direct supervision from reference responses.
We propose to decouple knowledge retrieval from response generation and introduce a multi-grained knowledge retriever.
arXiv Detail & Related papers (2023-05-17T12:12:46Z) - Open-Retrieval Conversational Question Answering [62.11228261293487]
We introduce an open-retrieval conversational question answering (ORConvQA) setting, where we learn to retrieve evidence from a large collection before extracting answers.
We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers.
arXiv Detail & Related papers (2020-05-22T19:39:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.