How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation

• URL: http://arxiv.org/abs/2608.25934v1
• Date: Wed, 26 Aug 2026 15:50:10 GMT
• Title: How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation
• Abstract summary: We cross-evaluate the full two-stage retrieve-then-verify pipeline across diverse datasets.<n>We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers.
• Score: 9.253244973042655
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.

Related papers

• PCQA-R1: Advancing Generalized 3D Point Cloud Quality Assessment with Reinforcement Learning [51.75061005702151]
  No-reference point cloud quality assessment (PCQA) has been an active topic in recent years and is used to measure and optimize the visual experience of point clouds.<n>We present PCQA-R1, the first reinforcement learning LMM for 3D point cloud quality assessment to simultaneously model quality understanding and scoring.
  arXiv  Detail & Related papers  (2026-08-19T07:21:58Z)
• Vision-Language Models for Analog Gauge Reading: An Empirical Study of Specialization, Transfer and Reliability [3.112410411428948]
  This study is a systematic evaluation of specialization, transfer, robustness and reliability for a general-purpose vision-language model (VLM)<n>The Qwen2.5-VL-7B-Instruct model is evaluated using zero-shot prompting, in-context learning (ICL) and parameter-efficient fine-tuning with Quantized Low-Rank Adaptation (QLoRA)<n>The best fine-tuned MPE values are 2.39% on the synthetic dataset, with a 95% confidence interval (CI) of 1.43-3.90%; 2.61% on the Pressure Gauge dataset, with a CI of 1.66-3.80%; and 4.4
  arXiv  Detail & Related papers  (2026-08-18T12:47:14Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• What Matters for Grocery Product Retrieval with Open Source Vision Language Models [0.0]
  We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge.<n>Switching from raw web-scrapes to filtered datasets delivers up to 16.6% accuracy gains.<n>State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs.
  arXiv  Detail & Related papers  (2026-05-18T08:20:13Z)
• VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering [53.662676566188175]
  A key bottleneck lies in the lack of public, large-scale, high-quality Scientific Visual Question Answering (SVQA) datasets.<n>We propose a verification-centric Generate-then-Verify framework that first generates QA pairs with figure-associated textual context.<n>We instantiate this framework to curate VeriSciQA, a dataset of 20,351 QA pairs spanning 20 scientific domains and 12 figure types.
  arXiv  Detail & Related papers  (2025-11-25T04:14:52Z)
• A Comprehensive Forecasting-Based Framework for Time Series Anomaly Detection: Benchmarking on the Numenta Anomaly Benchmark (NAB) [0.0]
  Time series anomaly detection is critical for modern digital infrastructures.<n>We present a forecasting-based framework unifying classical methods with deep learning architectures.<n>We conduct the first complete evaluation on the Numenta Anomaly Benchmark.
  arXiv  Detail & Related papers  (2025-10-13T08:31:42Z)
• NAIPv2: Debiased Pairwise Learning for Efficient Paper Quality Estimation [58.30936615525824]
  We present NAIPv2, a debiased and efficient framework for paper quality estimation.<n> NAIPv2 employs pairwise learning within domain-year groups to reduce inconsistencies in reviewer ratings.<n>It is trained on pairwise comparisons but enabling efficient pointwise prediction at deployment.
  arXiv  Detail & Related papers  (2025-09-29T17:59:23Z)
• Deepfake Detection that Generalizes Across Benchmarks [48.85953407706351]
  The generalization of deepfake detectors to unseen manipulation techniques remains a challenge for practical deployment.<n>This work demonstrates that robust generalization is achievable through a parameter-efficient adaptation of one of the foundational pre-trained vision encoders.<n>The proposed method achieves state-of-the-art performance, outperforming more complex, recent approaches in average cross-dataset AUROC.
  arXiv  Detail & Related papers  (2025-08-08T12:03:56Z)
• LLMEval-3: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models [51.55869466207234]
  Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting.<n>We introduce LLMEval-3, a framework for dynamic evaluation of LLMs.<n>LLEval-3 is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run.
  arXiv  Detail & Related papers  (2025-08-07T14:46:30Z)
• HyperspectralViTs: General Hyperspectral Models for On-board Remote Sensing [21.192836739734435]
  On-board processing of hyperspectral data with machine learning models would enable unprecedented amount of autonomy for a wide range of tasks.<n>This can enable early warning system and could allow new capabilities such as automated scheduling across constellations of satellites.<n>We propose fast and accurate machine learning architectures which support end-to-end training with data of high spectral dimension.
  arXiv  Detail & Related papers  (2024-10-22T17:59:55Z)
• Fact Checking with Insufficient Evidence [52.879658637466605]
  We study what information fact checking models consider sufficient by introducing a novel task and advancing it with three main contributions. We identify when models consider the remaining evidence (in)sufficient for FC, based on three trained models with different Transformer architectures. Second, we ask annotators whether the omitted evidence was important for FC, resulting in a novel diagnostic dataset, SufficientFacts, for FC with omitted evidence.
  arXiv  Detail & Related papers  (2022-04-05T06:12:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.