Fact Checking with Insufficient Evidence

• URL: http://arxiv.org/abs/2204.02007v1
• Date: Tue, 5 Apr 2022 06:12:42 GMT
• Title: Fact Checking with Insufficient Evidence
• Abstract summary: We study what information fact checking models consider sufficient by introducing a novel task and advancing it with three main contributions. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with different Transformer architectures. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset, SufficientFacts, for FC with omitted evidence.
• Score: 52.879658637466605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automating the fact checking (FC) process relies on information obtained from external sources. In this work, we posit that it is crucial for FC models to make veracity predictions only when there is sufficient evidence and otherwise indicate when it is not enough. To this end, we are the first to study what information FC models consider sufficient by introducing a novel task and advancing it with three main contributions. First, we conduct an in-depth empirical analysis of the task with a new fluency-preserving method for omitting information from the evidence at the constituent and sentence level. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with different Transformer architectures and three FC datasets. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset, SufficientFacts, for FC with omitted evidence. We find that models are least successful in detecting missing evidence when adverbial modifiers are omitted (21% accuracy), whereas it is easiest for omitted date modifiers (63% accuracy). Finally, we propose a novel data augmentation strategy for contrastive self-learning of missing evidence by employing the proposed omission method combined with tri-training. It improves performance for Evidence Sufficiency Prediction by up to 17.8 F1 score, which in turn improves FC performance by up to 2.6 F1 score.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Evian: Towards Explainable Visual Instruction-tuning Data Auditing [14.93566912726999]
  We introduce a novel "Decomposition-then-Evaluation" paradigm that breaks model responses into constituent cognitive components.<n>We instantiate this paradigm via EVIAN, an automated framework that evaluates these components along the axes of Image-Text Consistency, Logical Coherence, and Factual Accuracy.
  arXiv  Detail & Related papers  (2026-04-22T13:28:27Z)
• Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity [7.793738133162104]
  AMuFC is a multimodal fact-checking framework that employs two collaborative agents for the adaptive use of visual evidence.<n>We show that incorporating the Analyzer's assessment of visual evidence necessity into the Verifier's prediction yields substantial improvements in verification performance.<n>In addition to all code, we release WebFC, a newly constructed dataset for evaluating fact-checking modules in a more realistic scenario.
  arXiv  Detail & Related papers  (2026-04-06T14:01:38Z)
• Handling Missing Data in Downstream Tasks With Distribution-Preserving Guarantees [2.54990557236581]
  We propose an imputation approach named F3I based on the iterative improvement of a K-nearest neighbor imputation.<n>We provide a theoretical analysis of imputation quality and data distribution preservation by F3I for several types of missing mechanisms.<n>We demonstrate the superior performance of F3I on several imputation and classification tasks.
  arXiv  Detail & Related papers  (2025-01-23T16:04:18Z)
• Robust and Explainable Fine-Grained Visual Classification with Transfer Learning: A Dual-Carriageway Framework [0.799543372823325]
  We present an automatic best-suit training solution searching framework, the Dual-Carriageway Framework (DCF) We validated DCF's effectiveness through experiments with three convolutional neural networks (ResNet18, ResNet34 and Inception-v3) Results showed fine-tuning pathways outperformed training-from-scratch ones by up to 2.13% and 1.23% on the pre-existing and new datasets, respectively.
  arXiv  Detail & Related papers  (2024-05-09T15:41:10Z)
• Federated Causal Discovery from Heterogeneous Data [70.31070224690399]
  We propose a novel FCD method attempting to accommodate arbitrary causal models and heterogeneous data. These approaches involve constructing summary statistics as a proxy of the raw data to protect data privacy. We conduct extensive experiments on synthetic and real datasets to show the efficacy of our method.
  arXiv  Detail & Related papers  (2024-02-20T18:53:53Z)
• Heterogeneous Graph Reasoning for Fact Checking over Texts and Tables [22.18384189336634]
  HeterFC is a word-level Heterogeneous-graph-based model for Fact Checking over unstructured and structured information. We perform information propagation via a relational graph neural network, interactions between claims and evidence. We introduce a multitask loss function to account for potential inaccuracies in evidence retrieval.
  arXiv  Detail & Related papers  (2024-02-20T14:10:40Z)
• EntGPT: Linking Generative Large Language Models with Knowledge Bases [9.067856411512427]
  The ability of Large Language Models to generate factually correct output remains relatively unexplored. We design a three-step hard-prompting method to probe LLMs' ED performance without supervised fine-tuning. We further improve the knowledge grounding ability through instruction tuning (IT) with similar prompts and responses.
  arXiv  Detail & Related papers  (2024-02-09T19:16:27Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Improving the Adversarial Robustness of NLP Models by Information Bottleneck [112.44039792098579]
  Non-robust features can be easily manipulated by adversaries to fool NLP models. In this study, we explore the feasibility of capturing task-specific robust features, while eliminating the non-robust ones by using the information bottleneck theory. We show that the models trained with our information bottleneck-based method are able to achieve a significant improvement in robust accuracy.
  arXiv  Detail & Related papers  (2022-06-11T12:12:20Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• MDFM: Multi-Decision Fusing Model for Few-Shot Learning [16.47647579893923]
  We propose a novel method Multi-Decision Fusing Model (MDFM) to enhance the efficacy and robustness of the model. We evaluate the proposed method on five benchmark datasets and achieve significant improvements of 3.4%-7.3% compared with state-of-the-arts.
  arXiv  Detail & Related papers  (2021-12-01T18:13:09Z)
• FRITL: A Hybrid Method for Causal Discovery in the Presence of Latent Confounders [46.31784571870808]
  We show that under some mild assumptions, the model is uniquely identified by a hybrid method. Our method leverages the advantages of constraint-based methods and independent noise-based methods to handle both confounded and unconfounded situations.
  arXiv  Detail & Related papers  (2021-03-26T03:12:14Z)
• Dynamically Mitigating Data Discrepancy with Balanced Focal Loss for Replay Attack Detection [10.851348154870852]
  We argue that for anti-spoofing, it needs more attention for indistinguishable samples over easily-classified ones in the modeling process. We propose to leverage a balanced focal loss function as the training objective to dynamically scale the loss based on the traits of the sample itself. With complementary features, our fusion system with only three kinds of features outperforms other systems by 22.5% for min-tDCF and 7% for EER.
  arXiv  Detail & Related papers  (2020-06-25T17:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.