An Adversarial Benchmark for Fake News Detection Models

• URL: http://arxiv.org/abs/2201.00912v1
• Date: Mon, 3 Jan 2022 23:51:55 GMT
• Title: An Adversarial Benchmark for Fake News Detection Models
• Authors: Lorenzo Jaime Yu Flores, Yiding Hao
• Abstract summary: We formulate adversarial attacks that target three aspects of "understanding" We test our benchmark using BERT classifiers fine-tuned on the LIAR arXiv:arch-ive/1705648 and Kaggle Fake-News datasets.
• Score: 0.065268245109828
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the proliferation of online misinformation, fake news detection has gained importance in the artificial intelligence community. In this paper, we propose an adversarial benchmark that tests the ability of fake news detectors to reason about real-world facts. We formulate adversarial attacks that target three aspects of "understanding": compositional semantics, lexical relations, and sensitivity to modifiers. We test our benchmark using BERT classifiers fine-tuned on the LIAR arXiv:arch-ive/1705648 and Kaggle Fake-News datasets, and show that both models fail to respond to changes in compositional and lexical meaning. Our results strengthen the need for such models to be used in conjunction with other fact checking methods.

Related papers

• Adapting Fake News Detection to the Era of Large Language Models [48.5847914481222]
  We study the interplay between machine-(paraphrased) real news, machine-generated fake news, human-written fake news, and human-written real news. Our experiments reveal an interesting pattern that detectors trained exclusively on human-written articles can indeed perform well at detecting machine-generated fake news, but not vice versa.
  arXiv  Detail & Related papers  (2023-11-02T08:39:45Z)
• Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection [50.07850264495737]
  "Prompt-and-Align" (P&A) is a novel prompt-based paradigm for few-shot fake news detection. We show that P&A sets new states-of-the-art for few-shot fake news detection performance by significant margins.
  arXiv  Detail & Related papers  (2023-09-28T13:19:43Z)
• Verifying the Robustness of Automatic Credibility Assessment [50.55687778699995]
  We show that meaning-preserving changes in input text can mislead the models. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on misinformation detection tasks. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions.
  arXiv  Detail & Related papers  (2023-03-14T16:11:47Z)
• Nothing Stands Alone: Relational Fake News Detection with Hypergraph Neural Networks [49.29141811578359]
  We propose to leverage a hypergraph to represent group-wise interaction among news, while focusing on important news relations with its dual-level attention mechanism. Our approach yields remarkable performance and maintains the high performance even with a small subset of labeled news data.
  arXiv  Detail & Related papers  (2022-12-24T00:19:32Z)
• Adversarial Contrastive Learning for Evidence-aware Fake News Detection with Graph Neural Networks [20.282527436527765]
  We propose a unified Graph-based sEmantic structure mining framework with ConTRAstive Learning, namely GETRAL in short. We first model claims and evidences as graph-structured data to capture the long-distance semantic dependency. Then the fine-grained semantic representations are fed into the claim-evidence interaction module for predictions.
  arXiv  Detail & Related papers  (2022-10-11T14:54:37Z)
• A Multi-Policy Framework for Deep Learning-Based Fake News Detection [0.31498833540989407]
  This work introduces Multi-Policy Statement Checker (MPSC), a framework that automates fake news detection. MPSC uses deep learning techniques to analyze a statement itself and its related news articles, predicting whether it is seemingly credible or suspicious.
  arXiv  Detail & Related papers  (2022-06-01T21:25:21Z)
• Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation [105.20743048379387]
  We propose a novel framework for generating training examples informed by the known styles and strategies of human-authored propaganda. Specifically, we perform self-critical sequence training guided by natural language inference to ensure the validity of the generated articles. Our experimental results show that fake news detectors trained on PropaNews are better at detecting human-written disinformation by 3.62 - 7.69% F1 score on two public datasets.
  arXiv  Detail & Related papers  (2022-03-10T14:24:19Z)
• Mining Fine-grained Semantics via Graph Neural Networks for Evidence-based Fake News Detection [20.282527436527765]
  We propose a unified Graph-based sEmantic sTructure mining framework, namely GET in short. We model claims and evidences as graph-structured data and capture the long-distance semantic dependency. After obtaining contextual semantic information, our model reduces information redundancy by performing graph structure learning.
  arXiv  Detail & Related papers  (2022-01-18T11:28:36Z)
• Explainable Tsetlin Machine framework for fake news detection with credibility score assessment [16.457778420360537]
  We propose a novel interpretable fake news detection framework based on the recently introduced Tsetlin Machine (TM) We use the conjunctive clauses of the TM to capture lexical and semantic properties of both true and fake news text. For evaluation, we conduct experiments on two publicly available datasets, PolitiFact and GossipCop, and demonstrate that the TM framework significantly outperforms previously published baselines by at least $5%$ in terms of accuracy.
  arXiv  Detail & Related papers  (2021-05-19T13:18:02Z)
• Adversarial Active Learning based Heterogeneous Graph Neural Network for Fake News Detection [18.847254074201953]
  We propose a novel fake news detection framework, namely Adversarial Active Learning-based Heterogeneous Graph Neural Network (AA-HGNN) AA-HGNN utilizes an active learning framework to enhance learning performance, especially when facing the paucity of labeled data. Experiments with two real-world fake news datasets show that our model can outperform text-based models and other graph-based models.
  arXiv  Detail & Related papers  (2021-01-27T05:05:25Z)
• Machine Learning Explanations to Prevent Overtrust in Fake News Detection [64.46876057393703]
  This research investigates the effects of an Explainable AI assistant embedded in news review platforms for combating the propagation of fake news. We design a news reviewing and sharing interface, create a dataset of news stories, and train four interpretable fake news detection algorithms. For a deeper understanding of Explainable AI systems, we discuss interactions between user engagement, mental model, trust, and performance measures in the process of explaining.
  arXiv  Detail & Related papers  (2020-07-24T05:42:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.