Can Large Language Models Recall Reference Location Like Humans?
- URL: http://arxiv.org/abs/2402.17010v1
- Date: Mon, 26 Feb 2024 20:35:32 GMT
- Title: Can Large Language Models Recall Reference Location Like Humans?
- Authors: Ye Wang, Xinrun Xu, Rui Xie, Wenxin Hu, Wei Ye
- Abstract summary: This paper explores leveraging the parameterized knowledge stored during the pre-training phase of large language models to independently recall reference passage from any starting position.
Experiments on KILT knowledge-sensitive tasks have verified that LLMs can independently recall reference passage location in various task forms.
- Score: 8.657708519922002
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: When completing knowledge-intensive tasks, humans sometimes need not just an
answer but also a corresponding reference passage for auxiliary reading.
Previous methods required obtaining pre-segmented article chunks through
additional retrieval models. This paper explores leveraging the parameterized
knowledge stored during the pre-training phase of large language models (LLMs)
to independently recall reference passage from any starting position. We
propose a two-stage framework that simulates the scenario of humans recalling
easily forgotten references. Initially, the LLM is prompted to recall document
title identifiers to obtain a coarse-grained document set. Then, based on the
acquired coarse-grained document set, it recalls fine-grained passage. In the
two-stage recall process, we use constrained decoding to ensure that content
outside of the stored documents is not generated. To increase speed, we only
recall a short prefix in the second stage, then locate its position to retrieve
a complete passage. Experiments on KILT knowledge-sensitive tasks have verified
that LLMs can independently recall reference passage location in various task
forms, and the obtained reference significantly assist downstream tasks.
Related papers
- Identifying the Source of Generation for Large Language Models [21.919661430250798]
Large language models (LLMs) memorize text from several sources of documents.
LLMs can not provide document information on the generated content.
This work introduces token-level source identification in the decoding step.
arXiv Detail & Related papers (2024-07-05T08:52:15Z) - Ground Every Sentence: Improving Retrieval-Augmented LLMs with Interleaved Reference-Claim Generation [51.8188846284153]
RAG has been widely adopted to enhance Large Language Models (LLMs)
Attributed Text Generation (ATG) has attracted growing attention, which provides citations to support the model's responses in RAG.
This paper proposes a fine-grained ATG method called ReClaim(Refer & Claim), which alternates the generation of references and answers step by step.
arXiv Detail & Related papers (2024-07-01T20:47:47Z) - Memorizing Documents with Guidance in Large Language Models [21.919661430250798]
We propose document-wise memory architecture to track document memories in training.
We show that the proposed methods provide different memory entries for documents and high recall of document-related content in generation with trained document-wise memories.
arXiv Detail & Related papers (2024-06-23T03:12:03Z) - R4: Reinforced Retriever-Reorder-Responder for Retrieval-Augmented Large Language Models [32.598670876662375]
Retrieval-augmented large language models (LLMs) leverage relevant content retrieved by information retrieval systems to generate correct responses.
Existing retriever-responder methods typically append relevant documents to the prompt of LLMs to perform text generation tasks.
We propose a new pipeline named "Reinforced Retriever-Reorder-Responder" to learn document orderings for retrieval-augmented LLMs.
arXiv Detail & Related papers (2024-05-04T12:59:10Z) - Beyond Memorization: The Challenge of Random Memory Access in Language Models [56.525691003233554]
We investigate whether a generative Language Model (LM) is able to access its memory sequentially or randomly.
We find that techniques including recitation and permutation improve the random memory access capability of LMs.
arXiv Detail & Related papers (2024-03-12T16:42:44Z) - Previously on the Stories: Recap Snippet Identification for Story
Reading [51.641565531840186]
We propose the first benchmark on this useful task called Recap Snippet Identification with a hand-crafted evaluation dataset.
Our experiments show that the proposed task is challenging to PLMs, LLMs, and proposed methods as the task requires a deep understanding of the plot correlation between snippets.
arXiv Detail & Related papers (2024-02-11T18:27:14Z) - DAPR: A Benchmark on Document-Aware Passage Retrieval [57.45793782107218]
We propose and name this task emphDocument-Aware Passage Retrieval (DAPR)
While analyzing the errors of the State-of-The-Art (SoTA) passage retrievers, we find the major errors (53.5%) are due to missing document context.
Our created benchmark enables future research on developing and comparing retrieval systems for the new task.
arXiv Detail & Related papers (2023-05-23T10:39:57Z) - Large Language Models are Built-in Autoregressive Search Engines [19.928494069013485]
Large language models (LLMs) can follow human instructions to directly generate URLs for document retrieval.
LLMs can generate Web URLs where nearly 90% of the corresponding documents contain correct answers to open-domain questions.
arXiv Detail & Related papers (2023-05-16T17:04:48Z) - Recitation-Augmented Language Models [85.30591349383849]
We show that RECITE is a powerful paradigm for knowledge-intensive NLP tasks.
Specifically, we show that by utilizing recitation as the intermediate step, a recite-and-answer scheme can achieve new state-of-the-art performance.
arXiv Detail & Related papers (2022-10-04T00:49:20Z) - Generate rather than Retrieve: Large Language Models are Strong Context
Generators [74.87021992611672]
We present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators.
We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given question, and then reads the generated documents to produce the final answer.
arXiv Detail & Related papers (2022-09-21T01:30:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.