FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval

• URL: http://arxiv.org/abs/2410.21012v1
• Date: Mon, 28 Oct 2024 13:36:41 GMT
• Title: FACT: Examining the Effectiveness of Iterative Context Rewriting for Multi-fact Retrieval
• Authors: Jinlin Wang, Suyuchen Wang, Ziwen Xia, Sirui Hong, Yun Zhu, Bang Liu, Chenglin Wu,
• Abstract summary: Large Language Models (LLMs) are proficient at retrieving single facts from extended contexts, yet struggle with tasks requiring the simultaneous retrieval of multiple facts. This paper identifies a novel "lost-in-the-middle" phenomenon, where LLMs progressively lose track of critical information throughout the generation process. We introduce Find All Crucial Texts (FACT), an iterative retrieval method that refines context through successive rounds of rewriting.
• Score: 20.217386507637475
• License:
• Abstract: Large Language Models (LLMs) are proficient at retrieving single facts from extended contexts, yet they struggle with tasks requiring the simultaneous retrieval of multiple facts, especially during generation. This paper identifies a novel "lost-in-the-middle" phenomenon, where LLMs progressively lose track of critical information throughout the generation process, resulting in incomplete or inaccurate retrieval. To address this challenge, we introduce Find All Crucial Texts (FACT), an iterative retrieval method that refines context through successive rounds of rewriting. This approach enables models to capture essential facts incrementally, which are often overlooked in single-pass retrieval. Experiments demonstrate that FACT substantially enhances multi-fact retrieval performance across various tasks, though improvements are less notable in general-purpose QA scenarios. Our findings shed light on the limitations of LLMs in multi-fact retrieval and underscore the need for more resilient long-context retrieval strategies.

Related papers

• MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs [78.5013630951288]
  This paper introduces techniques for advancing information retrieval with multimodal large language models (MLLMs) We first study fine-tuning an MLLM as a bi-encoder retriever on 10 datasets with 16 retrieval tasks. We propose modality-aware hard negative mining to mitigate the modality bias exhibited by MLLM retrievers.
  arXiv  Detail & Related papers  (2024-11-04T20:06:34Z)
• Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data [6.195658947075431]
  We introduce HoloBench, a framework that brings database reasoning operations into text-based contexts. We show that the amount of information in the context has a bigger influence on LCLM performance than the context length. We find that tasks requiring the aggregation of multiple pieces of information show a noticeable drop in accuracy as context length increases.
  arXiv  Detail & Related papers  (2024-10-15T19:04:13Z)
• Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Context [31.091013417498825]
  We propose a simple yet effective method called context repetition (CoRe) CoRe involves prompting the model by repeatedly presenting the context to ensure the supporting documents are presented in the optimal order for the model. We improve the F1 score by up to 30%p on multi-hop QA tasks and increase accuracy by up to 70%p on a synthetic task.
  arXiv  Detail & Related papers  (2024-10-09T17:41:53Z)
• Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization [0.27624021966289597]
  This paper introduces EYEGLAXS, a framework that leverages Large Language Models (LLMs) for extractive summarization. EYEGLAXS focuses on extractive summarization to ensure factual and grammatical integrity. The system sets new performance benchmarks on well-known datasets like PubMed and ArXiv.
  arXiv  Detail & Related papers  (2024-08-28T13:52:19Z)
• Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment [16.39696580487218]
  Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. Recent research has explored using large language models (LLMs) as retrievers, achieving SOTA performance across various tasks.
  arXiv  Detail & Related papers  (2024-08-22T08:16:07Z)
• Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach [6.549143816134531]
  We propose a novel iterative RAG method called ReSP, equipped with a dual-function summarizer. Experimental results on the multi-hop question-answering HotpotQA and 2WikiMultihopQA demonstrate that our method significantly outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2024-07-18T02:19:00Z)
• Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation [51.8188846284153]
  RAG has been widely adopted to enhance Large Language Models (LLMs) Attributed Text Generation (ATG) has attracted growing attention, which provides citations to support the model's responses in RAG. This paper proposes a fine-grained ATG method called ReClaim(Refer & Claim), which alternates the generation of references and answers step by step.
  arXiv  Detail & Related papers  (2024-07-01T20:47:47Z)
• Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA [71.04146366608904]
  Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. We propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA) Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning.
  arXiv  Detail & Related papers  (2024-06-25T09:42:56Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)
• Modeling Uncertainty and Using Post-fusion as Fallback Improves Retrieval Augmented Generation with LLMs [80.74263278847063]
  The integration of retrieved passages and large language models (LLMs) has significantly contributed to improving open-domain question answering. This paper investigates different methods of combining retrieved passages with LLMs to enhance answer generation.
  arXiv  Detail & Related papers  (2023-08-24T05:26:54Z)
• Synergistic Interplay between Search and Large Language Models for Information Retrieval [141.18083677333848]
  InteR allows RMs to expand knowledge in queries using LLM-generated knowledge collections. InteR achieves overall superior zero-shot retrieval performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-12T11:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.