Uncertainty Quantification in Retrieval Augmented Question Answering

• URL: http://arxiv.org/abs/2502.18108v1
• Date: Tue, 25 Feb 2025 11:24:52 GMT
• Title: Uncertainty Quantification in Retrieval Augmented Question Answering
• Authors: Laura Perez-Beltrachini, Mirella Lapata,
• Abstract summary: We propose to quantify the uncertainty of a QA model via estimating the utility of the passages it is provided with.<n>We train a lightweight neural model to predict passage utility for a target QA model and show that while simple information theoretic metrics can predict answer correctness up to a certain extent, our approach efficiently approximates or outperforms more expensive sampling-based methods.
• Score: 57.05827081638329
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval augmented Question Answering (QA) helps QA models overcome knowledge gaps by incorporating retrieved evidence, typically a set of passages, alongside the question at test time. Previous studies show that this approach improves QA performance and reduces hallucinations, without, however, assessing whether the retrieved passages are indeed useful at answering correctly. In this work, we propose to quantify the uncertainty of a QA model via estimating the utility of the passages it is provided with. We train a lightweight neural model to predict passage utility for a target QA model and show that while simple information theoretic metrics can predict answer correctness up to a certain extent, our approach efficiently approximates or outperforms more expensive sampling-based methods. Code and data are available at https://github.com/lauhaide/ragu.

Related papers

• Uncertainty-aware Language Modeling for Selective Question Answering [107.47864420630923]
  We present an automatic large language model (LLM) conversion approach that produces uncertainty-aware LLMs. Our approach is model- and data-agnostic, is computationally-efficient, and does not rely on external models or systems.
  arXiv  Detail & Related papers  (2023-11-26T22:47:54Z)
• Improving Visual Question Answering Models through Robustness Analysis and In-Context Learning with a Chain of Basic Questions [70.70725223310401]
  This work proposes a new method that utilizes semantically related questions, referred to as basic questions, acting as noise to evaluate the robustness of VQA models. The experimental results demonstrate that the proposed evaluation method effectively analyzes the robustness of VQA models.
  arXiv  Detail & Related papers  (2023-04-06T15:32:35Z)
• Reinforced Question Rewriting for Conversational Question Answering [25.555372505026526]
  We develop a model to rewrite conversational questions into self-contained ones. It allows using existing single-turn QA systems to avoid training a CQA model from scratch. We propose using QA feedback to supervise the rewriting model with reinforcement learning.
  arXiv  Detail & Related papers  (2022-10-27T21:23:36Z)
• Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering [38.071375112873675]
  We propose a question-answer augmented encoder-decoder model and accompanying pretraining strategy. This yields an end-to-end system that outperforms prior QA retrieval methods on single-hop QA tasks.
  arXiv  Detail & Related papers  (2022-04-10T02:33:00Z)
• Counterfactual Variable Control for Robust and Interpretable Question Answering [57.25261576239862]
  Deep neural network based question answering (QA) models are neither robust nor explainable in many cases. In this paper, we inspect such spurious "capability" of QA models using causal inference. We propose a novel approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlation.
  arXiv  Detail & Related papers  (2020-10-12T10:09:05Z)
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA [95.9105154311491]
  We introduce two approaches to improve unsupervised Question Answering (QA) First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA) Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA.
  arXiv  Detail & Related papers  (2020-05-06T15:56:06Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.