How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models

• URL: http://arxiv.org/abs/2407.00369v1
• Date: Sat, 29 Jun 2024 08:39:07 GMT
• Title: How to Train Your Fact Verifier: Knowledge Transfer with Multimodal Open Models
• Authors: Jaeyoung Lee, Ximing Lu, Jack Hessel, Faeze Brahman, Youngjae Yu, Yonatan Bisk, Yejin Choi, Saadia Gabriel,
• Abstract summary: Large language or multimodal model based verification has been proposed to scale up online policing mechanisms for mitigating spread of false and harmful content. We test the limits of improving foundation model performance without continual updating through an initial study of knowledge transfer. Our results on two recent multi-modal fact-checking benchmarks, Mocheg and Fakeddit, indicate that knowledge transfer strategies can improve Fakeddit performance over the state-of-the-art by up to 1.7% and Mocheg performance by up to 2.9%.
• Score: 95.44559524735308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Given the growing influx of misinformation across news and social media, there is a critical need for systems that can provide effective real-time verification of news claims. Large language or multimodal model based verification has been proposed to scale up online policing mechanisms for mitigating spread of false and harmful content. While these can potentially reduce burden on human fact-checkers, such efforts may be hampered by foundation model training data becoming outdated. In this work, we test the limits of improving foundation model performance without continual updating through an initial study of knowledge transfer using either existing intra- and inter- domain benchmarks or explanations generated from large language models (LLMs). We evaluate on 12 public benchmarks for fact-checking and misinformation detection as well as two other tasks relevant to content moderation -- toxicity and stance detection. Our results on two recent multi-modal fact-checking benchmarks, Mocheg and Fakeddit, indicate that knowledge transfer strategies can improve Fakeddit performance over the state-of-the-art by up to 1.7% and Mocheg performance by up to 2.9%.

Related papers

• Detect, Investigate, Judge and Determine: A Novel LLM-based Framework for Few-shot Fake News Detection [47.01850264003063]
  Few-Shot Fake News Detection aims to distinguish inaccurate news from real ones in extremely low-resource scenarios. This task has garnered increased attention due to the widespread dissemination and harmful impact of fake news on social media. We propose a Dual-perspective Augmented Fake News Detection model, designed to enhance Large Language Models.
  arXiv  Detail & Related papers  (2024-07-12T03:15:01Z)
• Re-Search for The Truth: Multi-round Retrieval-augmented Large Language Models are Strong Fake News Detectors [38.75533934195315]
  Large Language Models (LLMs) are known for their remarkable reasoning and generative capabilities. We introduce a novel, retrieval-augmented LLMs framework--the first of its kind to automatically and strategically extract key evidence from web sources for claim verification. Our framework ensures the acquisition of sufficient, relevant evidence, thereby enhancing performance.
  arXiv  Detail & Related papers  (2024-03-14T00:35:39Z)
• Multimodal Large Language Models to Support Real-World Fact-Checking [80.41047725487645]
  Multimodal large language models (MLLMs) carry the potential to support humans in processing vast amounts of information. While MLLMs are already being used as a fact-checking tool, their abilities and limitations in this regard are understudied. We propose a framework for systematically assessing the capacity of current multimodal models to facilitate real-world fact-checking.
  arXiv  Detail & Related papers  (2024-03-06T11:32:41Z)
• Visual Explanations of Image-Text Representations via Multi-Modal Information Bottleneck Attribution [49.762034744605955]
  We propose a multi-modal information bottleneck approach to improve interpretability of vision-language models. We demonstrate how M2IB can be applied to attribution analysis of vision-language pretrained models.
  arXiv  Detail & Related papers  (2023-12-28T18:02:22Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Improving Generalization for Multimodal Fake News Detection [8.595270610973586]
  State-of-the-art approaches are usually trained on datasets of smaller size or with a limited set of specific topics. We propose three models that adopt and fine-tune state-of-the-art multimodal transformers for multimodal fake news detection.
  arXiv  Detail & Related papers  (2023-05-29T20:32:22Z)
• Leveraging Commonsense Knowledge on Classifying False News and Determining Checkworthiness of Claims [1.487444917213389]
  We propose to leverage commonsense knowledge for the tasks of false news classification and check-worthy claim detection. We fine-tune the BERT language model with a commonsense question answering task and the aforementioned tasks in a multi-task learning environment. Our experimental analysis demonstrates that commonsense knowledge can improve performance in both tasks.
  arXiv  Detail & Related papers  (2021-08-08T20:52:45Z)
• A Multi-Level Attention Model for Evidence-Based Fact Checking [58.95413968110558]
  We present a simple model that can be trained on sequence structures. Results on a large-scale dataset for Fact Extraction and VERification show that our model outperforms the graph-based approaches.
  arXiv  Detail & Related papers  (2021-06-02T05:40:12Z)
• Towards Few-Shot Fact-Checking via Perplexity [40.11397284006867]
  We propose a new way of utilizing the powerful transfer learning ability of a language model via a perplexity score. Our methodology can already outperform the Major Class baseline by more than absolute 10% on the F1-Macro metric. We construct and publicly release two new fact-checking datasets related to COVID-19.
  arXiv  Detail & Related papers  (2021-03-17T09:43:19Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.