Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models

• URL: http://arxiv.org/abs/2402.13492v3
• Date: Wed, 27 Mar 2024 18:48:34 GMT
• Title: Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models
• Authors: Seiji Maekawa, Hayate Iso, Sairam Gurajada, Nikita Bhutani,
• Abstract summary: We explore the effects of combinations of entities and relations on large language models (LMs) We observe that larger LMs excel in recalling popular facts, but encounter difficulty with infrequent entity-relation pairs compared to retrievers. We demonstrate the efficacy of our finer-grained metric and insights through an adaptive retrieval system.
• Score: 7.537599020279862
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While large language models (LMs) demonstrate remarkable performance, they encounter challenges in providing accurate responses when queried for information beyond their pre-trained memorization. Although augmenting them with relevant external information can mitigate these issues, failure to consider the necessity of retrieval may adversely affect overall performance. Previous research has primarily focused on examining how entities influence retrieval models and knowledge recall in LMs, leaving other aspects relatively unexplored. In this work, our goal is to offer a more detailed, fact-centric analysis by exploring the effects of combinations of entities and relations. To facilitate this, we construct a new question answering (QA) dataset called WiTQA (Wikipedia Triple Question Answers). This dataset includes questions about entities and relations of various popularity levels, each accompanied by a supporting passage. Our extensive experiments with diverse LMs and retrievers reveal when retrieval does not consistently enhance LMs from the viewpoints of fact-centric popularity. Confirming earlier findings, we observe that larger LMs excel in recalling popular facts. However, they notably encounter difficulty with infrequent entity-relation pairs compared to retrievers. Interestingly, they can effectively retain popular relations of less common entities. We demonstrate the efficacy of our finer-grained metric and insights through an adaptive retrieval system that selectively employs retrieval and recall based on the frequencies of entities and relations in the question.

Related papers

• Unraveling and Mitigating Retriever Inconsistencies in Retrieval-Augmented Large Language Models [25.346800371132645]
  Retrieval-augmented Large Language Models (RALMs) do not consistently outperform the original retrieval-free Language Models (LMs) Our experiments reveal that this example-level performance inconsistency exists not only between retrieval-augmented and retrieval-free LM but also among different retrievers. We introduce Ensemble of Retrievers (EoR), a trainable framework that can adaptively retrieve from different knowledge sources and effectively decrease unpredictable reader errors.
  arXiv  Detail & Related papers  (2024-05-31T08:22:49Z)
• Redefining Information Retrieval of Structured Database via Large Language Models [9.65171883231521]
  This paper introduces a novel retrieval augmentation framework called ChatLR. It primarily employs the powerful semantic understanding ability of Large Language Models (LLMs) as retrievers to achieve precise and concise information retrieval. Experimental results demonstrate the effectiveness of ChatLR in addressing user queries, achieving an overall information retrieval accuracy exceeding 98.8%.
  arXiv  Detail & Related papers  (2024-05-09T02:37:53Z)
• Optimizing Language Model's Reasoning Abilities with Weak Supervision [48.60598455782159]
  We present textscPuzzleBen, a weakly supervised benchmark that comprises 25,147 complex questions, answers, and human-generated rationales. A unique aspect of our dataset is the inclusion of 10,000 unannotated questions, enabling us to explore utilizing fewer supersized data to boost LLMs' inference capabilities.
  arXiv  Detail & Related papers  (2024-05-07T07:39:15Z)
• How Easily do Irrelevant Inputs Skew the Responses of Large Language Models? [35.78256134989427]
  Large Language Models (LLMs) exhibit enhanced capabilities for accomplishing many knowledge-intensive tasks. Due to the inherent flaws of current retrieval systems, there might exist irrelevant information within those retrieving top-ranked passages. We introduce a framework to construct high-quality irrelevant information that ranges from semantically unrelated, partially related, and related to questions.
  arXiv  Detail & Related papers  (2024-04-04T08:52:30Z)
• Reliable, Adaptable, and Attributable Language Models with Retrieval [144.26890121729514]
  Parametric language models (LMs) are trained on vast amounts of web data. They face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. We advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs.
  arXiv  Detail & Related papers  (2024-03-05T18:22:33Z)
• Discovery of the Hidden World with Large Language Models [100.38157787218044]
  We introduce COAT: Causal representatiOn AssistanT. COAT incorporates LLMs as a factor proposer that extracts the potential causal factors from unstructured data. LLMs can also be instructed to provide additional information used to collect data values.
  arXiv  Detail & Related papers  (2024-02-06T12:18:54Z)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection [74.51523859064802]
  We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) Self-RAG enhances an LM's quality and factuality through retrieval and self-reflection. It significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-10-17T18:18:32Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories [58.3421305091187]
  This paper aims to understand LMs' strengths and limitations in memorizing factual knowledge. We find that LMs struggle with less popular factual knowledge, and that scaling fails to appreciably improve memorization of factual knowledge in the long tail. We devise a simple, yet effective, method for powerful and efficient retrieval-augmented LMs, which retrieves non-parametric memories only when necessary.
  arXiv  Detail & Related papers  (2022-12-20T18:30:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.