DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts

• URL: http://arxiv.org/abs/2412.10510v2
• Date: Thu, 06 Feb 2025 13:27:38 GMT
• Title: DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts
• Authors: Tobias Braun, Mark Rothermel, Marcus Rohrbach, Anna Rohrbach,
• Abstract summary: Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME) is a zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME operates in a six-stage process, dynamically selecting the tools and search depth to extract and evaluate textual and visual evidence. Evaluation on the popular benchmarks VERITE, AVerITeC, and MOCHEG shows that DEFAME surpasses all previous methods.
• Score: 35.952854524873246
• License:
• Abstract: The proliferation of disinformation demands reliable and scalable fact-checking solutions. We present Dynamic Evidence-based FAct-checking with Multimodal Experts (DEFAME), a modular, zero-shot MLLM pipeline for open-domain, text-image claim verification. DEFAME operates in a six-stage process, dynamically selecting the tools and search depth to extract and evaluate textual and visual evidence. Unlike prior approaches that are text-only, lack explainability, or rely solely on parametric knowledge, DEFAME performs end-to-end verification, accounting for images in claims and evidence while generating structured, multimodal reports. Evaluation on the popular benchmarks VERITE, AVerITeC, and MOCHEG shows that DEFAME surpasses all previous methods, establishing itself as the new state-of-the-art fact-checking system for uni- and multimodal fact-checking. Moreover, we introduce a new benchmark, CLAIMREVIEW24+, featuring claims after the knowledge cutoff of GPT4o to avoid data leakage. Here, DEFAME drastically outperforms the GPT Chain-of-Thought baseline, demonstrating temporal generalizability and the potential for real-time fact-checking.

Related papers

• VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation [100.06122876025063]
  This paper introduces VisDoMBench, the first comprehensive benchmark designed to evaluate QA systems in multi-document settings. We propose VisDoMRAG, a novel multimodal Retrieval Augmented Generation (RAG) approach that simultaneously utilizes visual and textual RAG.
  arXiv  Detail & Related papers  (2024-12-14T06:24:55Z)
• FIRE: Fact-checking with Iterative Retrieval and Verification [63.67320352038525]
  FIRE is a novel framework that integrates evidence retrieval and claim verification in an iterative manner. It achieves slightly better performance while reducing large language model (LLM) costs by an average of 7.6 times and search costs by 16.5 times. These results indicate that FIRE holds promise for application in large-scale fact-checking operations.
  arXiv  Detail & Related papers  (2024-10-17T06:44:18Z)
• Multimodal Misinformation Detection using Large Vision-Language Models [7.505532091249881]
  Large language models (LLMs) have shown remarkable performance in various tasks. Few approaches consider evidence retrieval as part of misinformation detection. We propose a novel re-ranking approach for multimodal evidence retrieval.
  arXiv  Detail & Related papers  (2024-07-19T13:57:11Z)
• MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking [0.283600654802951]
  We present a summarization model designed to generate claim-specific summaries useful for fact-checking from multimodal datasets. We introduce a dynamic perceiver-based model that can handle inputs from multiple modalities of arbitrary lengths. Our approach outperforms the SOTA approach by 4.6% in the claim verification task on the MOCHEG dataset.
  arXiv  Detail & Related papers  (2024-07-18T01:33:20Z)
• Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers [121.53749383203792]
  We present a holistic end-to-end solution for annotating the factuality of large language models (LLMs)-generated responses. We construct an open-domain document-level factuality benchmark in three-level granularity: claim, sentence and document. Preliminary experiments show that FacTool, FactScore and Perplexity are struggling to identify false claims.
  arXiv  Detail & Related papers  (2023-11-15T14:41:57Z)
• WiCE: Real-World Entailment for Claims in Wikipedia [63.234352061821625]
  We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim. We show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.
  arXiv  Detail & Related papers  (2023-03-02T17:45:32Z)
• WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning [40.5830891229718]
  We propose a weakly supervised framework that aggregates multiple resources to train a precise and efficient factual metric, namely WeCheck. Comprehensive experiments on a variety of tasks demonstrate the strong performance of WeCheck, which achieves a 3.4% absolute improvement over previous state-of-the-art methods on TRUE benchmark on average.
  arXiv  Detail & Related papers  (2022-12-20T08:04:36Z)
• End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models [0.0]
  We propose end-to-end multimodal fact-checking and explanation generation. The goal is to assess the truthfulness of a claim by retrieving relevant evidence and predicting a truthfulness label. To support this research, we construct Mocheg, a large-scale dataset consisting of 15,601 claims.
  arXiv  Detail & Related papers  (2022-05-25T04:36:46Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• Logically at the Factify 2022: Multimodal Fact Verification [2.8914815569249823]
  This paper describes our participant system for the multi-modal fact verification (Factify) challenge at AAAI 2022. Two baseline approaches are proposed and explored including an ensemble model and a multi-modal attention network. Our best model is ranked first in leaderboard which obtains a weighted average F-measure of 0.77 on both validation and test set.
  arXiv  Detail & Related papers  (2021-12-16T23:34:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.