Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition

• URL: http://arxiv.org/abs/2409.17073v4
• Date: Sat, 23 Nov 2024 19:07:10 GMT
• Title: Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition
• Authors: Pritika Ramu, Koustava Goswami, Apoorv Saxena, Balaji Vasan Srinivasan,
• Abstract summary: Post-hoc attribution systems are designed to map answer text back to the source document, yet the granularity of this mapping has not been addressed. We propose and investigate a novel approach to the factual decomposition of generated answers for attribution, employing template-based in-context learning.
• Score: 10.585679421637948
• License:
• Abstract: Accurately attributing answer text to its source document is crucial for developing a reliable question-answering system. However, attribution for long documents remains largely unexplored. Post-hoc attribution systems are designed to map answer text back to the source document, yet the granularity of this mapping has not been addressed. Furthermore, a critical question arises: What exactly should be attributed? This involves identifying the specific information units within an answer that require grounding. In this paper, we propose and investigate a novel approach to the factual decomposition of generated answers for attribution, employing template-based in-context learning. To accomplish this, we utilize the question and integrate negative sampling during few-shot in-context learning for decomposition. This approach enhances the semantic understanding of both abstractive and extractive answers. We examine the impact of answer decomposition by providing a thorough examination of various attribution approaches, ranging from retrieval-based techniques to LLM-based attributors.

Related papers

• Enhancing Answer Attribution for Faithful Text Generation with Large Language Models [5.065947993017158]
  We propose new methods for producing more independent and contextualized claims for better retrieval and attribution. New methods are evaluated and shown to improve the performance of answer attribution components.
  arXiv  Detail & Related papers  (2024-10-22T15:37:46Z)
• Contri(e)ve: Context + Retrieve for Scholarly Question Answering [0.0]
  We present a two step solution using open source Large Language Model(LLM): Llama3.1 for Scholarly-QALD dataset. Firstly, we extract the context pertaining to the question from different structured and unstructured data sources. Secondly, we implement prompt engineering to improve the information retrieval performance of the LLM.
  arXiv  Detail & Related papers  (2024-09-13T17:38:47Z)
• Attribute or Abstain: Large Language Models as Long Document Assistants [58.32043134560244]
  LLMs can help humans working with long documents, but are known to hallucinate. Existing approaches to attribution have only been evaluated in RAG settings, where the initial retrieval confounds LLM performance. This is crucially different from the long document setting, where retrieval is not needed, but could help. We present LAB, a benchmark of 6 diverse long document tasks with attribution, and experiments with different approaches to attribution on 5 LLMs of different sizes.
  arXiv  Detail & Related papers  (2024-07-10T16:16:02Z)
• Learning to Filter Context for Retrieval-Augmented Generation [75.18946584853316]
  Generation models are required to generate outputs given partially or entirely irrelevant passages. FILCO identifies useful context based on lexical and information-theoretic approaches. It trains context filtering models that can filter retrieved contexts at test time.
  arXiv  Detail & Related papers  (2023-11-14T18:41:54Z)
• SEMQA: Semi-Extractive Multi-Source Question Answering [94.04430035121136]
  We introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. We create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions.
  arXiv  Detail & Related papers  (2023-11-08T18:46:32Z)
• DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text [73.68051228972024]
  Large Language Models (LLMs) have exhibited impressive generation capabilities, but they suffer from hallucinations when relying on their internal knowledge. Retrieval-augmented LLMs have emerged as a potential solution to ground LLMs in external knowledge.
  arXiv  Detail & Related papers  (2023-10-31T04:37:57Z)
• Understanding Retrieval Augmentation for Long-Form Question Answering [44.19142029392175]
  We present a study of retrieval-augmented language models (LMs) on long-form question answering. We analyze how retrieval augmentation impacts different LMs, by comparing answers generated from models while using the same evidence documents.
  arXiv  Detail & Related papers  (2023-10-18T17:59:10Z)
• Enhance Long Text Understanding via Distilled Gist Detector from Abstractive Summarization [7.851265919027389]
  We consider the problem of how to disentangle the gist-relevant and irrelevant information for long text understanding. Experiments on document classification, distantly supervised open-domain question answering (DS-QA) and non-parallel text style transfer show that our method can significantly improve the performance of the baseline models.
  arXiv  Detail & Related papers  (2021-10-10T09:21:24Z)
• Discrete Reasoning Templates for Natural Language Understanding [79.07883990966077]
  We present an approach that reasons about complex questions by decomposing them to simpler subquestions. We derive the final answer according to instructions in a predefined reasoning template. We show that our approach is competitive with the state-of-the-art while being interpretable and requires little supervision.
  arXiv  Detail & Related papers  (2021-04-05T18:56:56Z)
• Knowledge-Aided Open-Domain Question Answering [58.712857964048446]
  We propose a knowledge-aided open-domain QA (KAQA) method which targets at improving relevant document retrieval and answer reranking. During document retrieval, a candidate document is scored by considering its relationship to the question and other documents. During answer reranking, a candidate answer is reranked using not only its own context but also the clues from other documents.
  arXiv  Detail & Related papers  (2020-06-09T13:28:57Z)
• Natural language processing for word sense disambiguation and information extraction [0.0]
  The thesis presents a new approach for Word Sense Disambiguation using thesaurus. A Document Retrieval method, based on Fuzzy Logic has been described and its application is illustrated. The strategy concludes with the presentation of a novel strategy based on Dempster-Shafer theory of evidential reasoning.
  arXiv  Detail & Related papers  (2020-04-05T17:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.