Explaining Sources of Uncertainty in Automated Fact-Checking

• URL: http://arxiv.org/abs/2505.17855v1
• Date: Fri, 23 May 2025 13:06:43 GMT
• Title: Explaining Sources of Uncertainty in Automated Fact-Checking
• Authors: Jingyi Sun, Greta Warren, Irina Shklovski, Isabelle Augenstein,
• Abstract summary: CLUE (Conflict-and-Agreement-aware Language-model Uncertainty Explanations) is a framework to generate natural language explanations of model uncertainty.<n>It identifies relationships between spans of text that expose claim-evidence or inter-evidence conflicts and agreements that drive the model's predictive uncertainty.<n> CLUE produces explanations that are more faithful to the model's uncertainty and more consistent with fact-checking decisions.
• Score: 41.236833314783134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding sources of a model's uncertainty regarding its predictions is crucial for effective human-AI collaboration. Prior work proposes using numerical uncertainty or hedges ("I'm not sure, but ..."), which do not explain uncertainty that arises from conflicting evidence, leaving users unable to resolve disagreements or rely on the output. We introduce CLUE (Conflict-and-Agreement-aware Language-model Uncertainty Explanations), the first framework to generate natural language explanations of model uncertainty by (i) identifying relationships between spans of text that expose claim-evidence or inter-evidence conflicts and agreements that drive the model's predictive uncertainty in an unsupervised way, and (ii) generating explanations via prompting and attention steering that verbalize these critical interactions. Across three language models and two fact-checking datasets, we show that CLUE produces explanations that are more faithful to the model's uncertainty and more consistent with fact-checking decisions than prompting for uncertainty explanations without span-interaction guidance. Human evaluators judge our explanations to be more helpful, more informative, less redundant, and more logically consistent with the input than this baseline. CLUE requires no fine-tuning or architectural changes, making it plug-and-play for any white-box language model. By explicitly linking uncertainty to evidence conflicts, it offers practical support for fact-checking and generalises readily to other tasks that require reasoning over complex information.

Related papers

• Conflicts in Texts: Data, Implications and Challenges [58.03478157713084]
  Conflicts could reflect the complexity of situations, changes that need to be explained and dealt with, difficulties in data annotation, and mistakes in generated outputs.<n>This survey categorizes these conflicts into three key areas: (1) natural texts on the web, where factual inconsistencies, subjective biases, and multiple perspectives introduce contradictions; (2) human-annotated data, where annotator disagreements, mistakes, and societal biases impact model training; and (3) model interactions, where hallucinations and knowledge conflicts emerge during deployment.<n>We highlight key challenges and future directions for developing conflict-aware NLP systems that can reason over and reconcile conflicting information more effectively
  arXiv  Detail & Related papers  (2025-04-28T04:24:01Z)
• CRAVE: A Conflicting Reasoning Approach for Explainable Claim Verification Using LLMs [15.170312674645535]
  CRAVE is a Conflicting Reasoning Approach for explainable claim VErification.<n>It can verify complex claims based on the conflicting rationales reasoned by large language models.<n>CRAVE achieves much better performance than state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-21T07:20:31Z)
• Identifying Drivers of Predictive Aleatoric Uncertainty [2.5311562666866494]
  We propose a straightforward approach to explain predictive aleatoric uncertainties.<n>We estimate uncertainty in regression as predictive variance by adapting a neural network with a Gaussian output distribution.<n>This approach can explain uncertainty influences more reliably than complex published approaches.
  arXiv  Detail & Related papers  (2023-12-12T13:28:53Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Improving the Reliability of Large Language Models by Leveraging Uncertainty-Aware In-Context Learning [76.98542249776257]
  Large-scale language models often face the challenge of "hallucination" We introduce an uncertainty-aware in-context learning framework to empower the model to enhance or reject its output in response to uncertainty.
  arXiv  Detail & Related papers  (2023-10-07T12:06:53Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Towards Improving Faithfulness in Abstractive Summarization [37.19777407790153]
  We propose a Faithfulness Enhanced Summarization model (FES) to improve fidelity in abstractive summarization. Our model outperforms strong baselines in experiments on CNN/DM and XSum.
  arXiv  Detail & Related papers  (2022-10-04T19:52:09Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Reliable Post hoc Explanations: Modeling Uncertainty in Explainability [44.9824285459365]
  Black box explanations are increasingly being employed to establish model credibility in high-stakes settings. prior work demonstrates that explanations generated by state-of-the-art techniques are inconsistent, unstable, and provide very little insight into their correctness and reliability. We develop a novel Bayesian framework for generating local explanations along with their associated uncertainty.
  arXiv  Detail & Related papers  (2020-08-11T22:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.