AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web

• URL: http://arxiv.org/abs/2505.17978v1
• Date: Fri, 23 May 2025 14:45:48 GMT
• Title: AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web
• Authors: Rui Cao, Zifeng Ding, Zhijiang Guo, Michael Schlichtkrull, Andreas Vlachos,
• Abstract summary: We introduce AVerImaTeC, a dataset consisting of 1,297 real-world image-text claims.<n>Each claim is annotated with question-answer pairs containing evidence from the web.<n>We mitigate challenges in fact-checking datasets such as contextual dependence, temporal leakage, and evidence insufficiency.
• Score: 25.513968401608924
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Textual claims are often accompanied by images to enhance their credibility and spread on social media, but this also raises concerns about the spread of misinformation. Existing datasets for automated verification of image-text claims remain limited, as they often consist of synthetic claims and lack evidence annotations to capture the reasoning behind the verdict. In this work, we introduce AVerImaTeC, a dataset consisting of 1,297 real-world image-text claims. Each claim is annotated with question-answer (QA) pairs containing evidence from the web, reflecting a decomposed reasoning regarding the verdict. We mitigate common challenges in fact-checking datasets such as contextual dependence, temporal leakage, and evidence insufficiency, via claim normalization, temporally constrained evidence annotation, and a two-stage sufficiency check. We assess the consistency of the annotation in AVerImaTeC via inter-annotator studies, achieving a $\kappa=0.742$ on verdicts and $74.7\%$ consistency on QA pairs. We also propose a novel evaluation method for evidence retrieval and conduct extensive experiments to establish baselines for verifying image-text claims using open-web evidence.

Related papers

• Contrastive Learning to Improve Retrieval for Real-world Fact Checking [84.57583869042791]
  We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for fact-checking complex claims. We leverage the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents. We find a 6% improvement in veracity classification accuracy on the dataset.
  arXiv  Detail & Related papers  (2024-10-07T00:09:50Z)
• Navigating the Noisy Crowd: Finding Key Information for Claim Verification [19.769771741059408]
  We propose EACon, a framework designed to find key information within evidence and verify each aspect of a claim separately. Eccon finds keywords from the claim and employs fuzzy matching to select relevant keywords for each raw evidence piece. Eccon deconstructs the original claim into subclaims, which are then verified against both abstracted and raw evidence individually.
  arXiv  Detail & Related papers  (2024-07-17T09:24:10Z)
• Give Me More Details: Improving Fact-Checking with Latent Retrieval [58.706972228039604]
  Evidence plays a crucial role in automated fact-checking. Existing fact-checking systems either assume the evidence sentences are given or use the search snippets returned by the search engine. We propose to incorporate full text from source documents as evidence and introduce two enriched datasets.
  arXiv  Detail & Related papers  (2023-05-25T15:01:19Z)
• AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web [20.576644330553744]
  We introduce AVeriTeC, a new dataset of 4,568 real-world claims covering fact-checks by 50 different organizations. Each claim is annotated with question-answer pairs supported by evidence available online, as well as textual justifications explaining how the evidence combines to produce a verdict.
  arXiv  Detail & Related papers  (2023-05-22T15:17:18Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)
• Generating Literal and Implied Subquestions to Fact-check Complex Claims [64.81832149826035]
  We focus on decomposing a complex claim into a comprehensive set of yes-no subquestions whose answers influence the veracity of the claim. We present ClaimDecomp, a dataset of decompositions for over 1000 claims. We show that these subquestions can help identify relevant evidence to fact-check the full claim and derive the veracity through their answers.
  arXiv  Detail & Related papers  (2022-05-14T00:40:57Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• COVID-Fact: Fact Extraction and Verification of Real-World Claims on COVID-19 Pandemic [12.078052727772718]
  We introduce a FEVER-like dataset COVID-Fact of $4,086$ claims concerning the COVID-19 pandemic. The dataset contains claims, evidence for the claims, and contradictory claims refuted by the evidence.
  arXiv  Detail & Related papers  (2021-06-07T16:59:46Z)
• AmbiFC: Fact-Checking Ambiguous Claims with Evidence [57.7091560922174]
  We present AmbiFC, a fact-checking dataset with 10k claims derived from real-world information needs. We analyze disagreements arising from ambiguity when comparing claims against evidence in AmbiFC. We develop models for predicting veracity handling this ambiguity via soft labels.
  arXiv  Detail & Related papers  (2021-04-01T17:40:08Z)
• Hierarchical Evidence Set Modeling for Automated Fact Extraction and Verification [5.836068916903788]
  Hierarchical Evidence Set Modeling (HESM) is a framework to extract evidence sets and verify a claim to be supported, refuted or not enough info. Our experimental results show that HESM outperforms 7 state-of-the-art methods for fact extraction and claim verification.
  arXiv  Detail & Related papers  (2020-10-10T22:27:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.