Self-Retrieval: Building an Information Retrieval System with One Large
Language Model
- URL: http://arxiv.org/abs/2403.00801v1
- Date: Fri, 23 Feb 2024 18:45:35 GMT
- Title: Self-Retrieval: Building an Information Retrieval System with One Large
Language Model
- Authors: Qiaoyu Tang, Jiawei Chen, Bowen Yu, Yaojie Lu, Cheng Fu, Haiyang Yu,
Hongyu Lin, Fei Huang, Ben He, Xianpei Han, Le Sun, Yongbin Li
- Abstract summary: Self-Retrieval is an end-to-end, LLM-driven information retrieval architecture.
We show that Self-Retrieval significantly outperforms previous retrieval approaches by a large margin.
- Score: 102.78988790457004
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The rise of large language models (LLMs) has transformed the role of
information retrieval (IR) systems in the way to humans accessing information.
Due to the isolated architecture and the limited interaction, existing IR
systems are unable to fully accommodate the shift from directly providing
information to humans to indirectly serving large language models. In this
paper, we propose Self-Retrieval, an end-to-end, LLM-driven information
retrieval architecture that can fully internalize the required abilities of IR
systems into a single LLM and deeply leverage the capabilities of LLMs during
IR process. Specifically, Self-retrieval internalizes the corpus to retrieve
into a LLM via a natural language indexing architecture. Then the entire
retrieval process is redefined as a procedure of document generation and
self-assessment, which can be end-to-end executed using a single large language
model. Experimental results demonstrate that Self-Retrieval not only
significantly outperforms previous retrieval approaches by a large margin, but
also can significantly boost the performance of LLM-driven downstream
applications like retrieval augumented generation.
Related papers
- IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues [10.280113107290067]
The IM-RAG approach integrates Information Retrieval systems with Large Language Models (LLMs) to support multi-round RAG.
The entire IM process is optimized via Reinforcement Learning (RL) where a Progress Tracker is incorporated to provide mid-step rewards.
The results show that our approach achieves state-of-the-art (SOTA) performance while providing high flexibility in integrating IR modules.
arXiv Detail & Related papers (2024-05-15T12:41:20Z) - Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks.
The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human.
These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
arXiv Detail & Related papers (2024-01-30T17:38:54Z) - INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning [59.07490387145391]
Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks.
Their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language.
We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories.
arXiv Detail & Related papers (2024-01-12T12:10:28Z) - Adapting LLMs for Efficient, Personalized Information Retrieval: Methods
and Implications [0.7832189413179361]
Large Language Models (LLMs) excel in comprehending and generating human-like text.
This paper explores strategies for integrating Language Models (LLMs) with Information Retrieval (IR) systems.
arXiv Detail & Related papers (2023-11-21T02:01:01Z) - Recommender AI Agent: Integrating Large Language Models for Interactive
Recommendations [53.76682562935373]
We introduce an efficient framework called textbfInteRecAgent, which employs LLMs as the brain and recommender models as tools.
InteRecAgent achieves satisfying performance as a conversational recommender system, outperforming general-purpose LLMs.
arXiv Detail & Related papers (2023-08-31T07:36:44Z) - Large Language Models for Information Retrieval: A Survey [57.7992728506871]
Information retrieval has evolved from term-based methods to its integration with advanced neural models.
Recent research has sought to leverage large language models (LLMs) to improve IR systems.
We delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers.
arXiv Detail & Related papers (2023-08-14T12:47:22Z) - Synergistic Interplay between Search and Large Language Models for
Information Retrieval [141.18083677333848]
InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections.
InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
arXiv Detail & Related papers (2023-05-12T11:58:15Z) - In-Context Retrieval-Augmented Language Models [28.23702459322163]
We show that In-Context RALM builds on off-the-shelf general purpose retrievers to provide surprisingly large LM gains across model sizes and diverse corpora.
We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding.
arXiv Detail & Related papers (2023-01-31T20:26:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.