Localizing and Mitigating Errors in Long-form Question Answering

• URL: http://arxiv.org/abs/2407.11930v3
• Date: Mon, 4 Nov 2024 10:30:16 GMT
• Title: Localizing and Mitigating Errors in Long-form Question Answering
• Authors: Rachneet Sachdeva, Yixiao Song, Mohit Iyyer, Iryna Gurevych,
• Abstract summary: Long-form question answering (LFQA) aims to provide thorough and in-depth answers to complex questions, enhancing comprehension. This work introduces HaluQuestQA, the first hallucination dataset with localized error annotations for human-written and model-generated LFQA answers.
• Score: 79.63372684264921
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Long-form question answering (LFQA) aims to provide thorough and in-depth answers to complex questions, enhancing comprehension. However, such detailed responses are prone to hallucinations and factual inconsistencies, challenging their faithful evaluation. This work introduces HaluQuestQA, the first hallucination dataset with localized error annotations for human-written and model-generated LFQA answers. HaluQuestQA comprises 698 QA pairs with 1.8k span-level error annotations for five different error types by expert annotators, along with preference judgments. Using our collected data, we thoroughly analyze the shortcomings of long-form answers and find that they lack comprehensiveness and provide unhelpful references. We train an automatic feedback model on this dataset that predicts error spans with incomplete information and provides associated explanations. Finally, we propose a prompt-based approach, Error-informed refinement, that uses signals from the learned feedback model to refine generated answers, which we show reduces errors and improves answer quality across multiple models. Furthermore, humans find answers generated by our approach comprehensive and highly prefer them (84%) over the baseline answers.

Related papers

• I Could've Asked That: Reformulating Unanswerable Questions [89.93173151422636]
  We evaluate open-source and proprietary models for reformulating unanswerable questions. GPT-4 and Llama2-7B successfully reformulate questions only 26% and 12% of the time, respectively. We publicly release the benchmark and the code to reproduce the experiments.
  arXiv  Detail & Related papers  (2024-07-24T17:59:07Z)
• Long-form Question Answering: An Iterative Planning-Retrieval-Generation Approach [28.849548176802262]
  Long-form question answering (LFQA) poses a challenge as it involves generating detailed answers in the form of paragraphs. We propose an LFQA model with iterative Planning, Retrieval, and Generation. We find that our model outperforms the state-of-the-art models on various textual and factual metrics for the LFQA task.
  arXiv  Detail & Related papers  (2023-11-15T21:22:27Z)
• Chain-of-Verification Reduces Hallucination in Large Language Models [80.99318041981776]
  We study the ability of language models to deliberate on the responses they give in order to correct their mistakes. We develop the Chain-of-Verification (CoVe) method whereby the model first drafts an initial response. We show CoVe decreases hallucinations across a variety of tasks, from list-based questions from Wikidata to closed book MultiSpanQA.
  arXiv  Detail & Related papers  (2023-09-20T17:50:55Z)
• Getting MoRE out of Mixture of Language Model Reasoning Experts [71.61176122960464]
  We propose a Mixture-of-Reasoning-Experts (MoRE) framework that ensembles diverse specialized language models. We specialize the backbone language model with prompts optimized for different reasoning categories, including factual, multihop, mathematical, and commonsense reasoning. Our human study confirms that presenting expert predictions and the answer selection process helps annotators more accurately calibrate when to trust the system's output.
  arXiv  Detail & Related papers  (2023-05-24T02:00:51Z)
• Model Analysis & Evaluation for Ambiguous Question Answering [0.0]
  Question Answering models are required to generate long-form answers that often combine conflicting pieces of information. Recent advances in the field have shown strong capabilities in generating fluent responses, but certain research questions remain unanswered. We aim to thoroughly investigate these aspects, and provide valuable insights into the limitations of the current approaches.
  arXiv  Detail & Related papers  (2023-05-21T15:20:20Z)
• Teaching language models to support answers with verified quotes [12.296242080730831]
  We train "open-book" QA models that generate answers whilst also citing specific evidence for their claims. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure.
  arXiv  Detail & Related papers  (2022-03-21T17:26:29Z)
• AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization [73.91543616777064]
  Community Question Answering (CQA) fora such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. This work introduces a novel dataset of 4,631 CQA threads for answer summarization, curated by professional linguists.
  arXiv  Detail & Related papers  (2021-11-11T21:48:02Z)
• Hurdles to Progress in Long-form Question Answering [34.805039943215284]
  We show that the task formulation raises fundamental challenges regarding evaluation and dataset creation. We first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-03-10T20:32:30Z)
• A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering [15.355557454305776]
  We show that question rewriting (QR) of the conversational context allows to shed more light on this phenomenon. We present the results of this analysis on the TREC CAsT and QuAC (CANARD) datasets.
  arXiv  Detail & Related papers  (2020-10-13T06:29:51Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.