Examining the Limitations of Computational Rumor Detection Models Trained on Static Datasets

• URL: http://arxiv.org/abs/2309.11576v2
• Date: Sun, 24 Mar 2024 00:06:49 GMT
• Title: Examining the Limitations of Computational Rumor Detection Models Trained on Static Datasets
• Authors: Yida Mu, Xingyi Song, Kalina Bontcheva, Nikolaos Aletras,
• Abstract summary: This paper is in-depth evaluation of the performance gap between content and context-based models. Our empirical findings demonstrate that context-based models are still overly dependent on the information derived from the rumors' source post. Based on our experimental results, the paper also offers practical suggestions on how to minimize the effects of temporal concept drift in static datasets.
• Score: 30.315424983805087
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: A crucial aspect of a rumor detection model is its ability to generalize, particularly its ability to detect emerging, previously unknown rumors. Past research has indicated that content-based (i.e., using solely source posts as input) rumor detection models tend to perform less effectively on unseen rumors. At the same time, the potential of context-based models remains largely untapped. The main contribution of this paper is in the in-depth evaluation of the performance gap between content and context-based models specifically on detecting new, unseen rumors. Our empirical findings demonstrate that context-based models are still overly dependent on the information derived from the rumors' source post and tend to overlook the significant role that contextual information can play. We also study the effect of data split strategies on classifier performance. Based on our experimental results, the paper also offers practical suggestions on how to minimize the effects of temporal concept drift in static datasets during the training of rumor detection methods.

Related papers

• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• A Unified Contrastive Transfer Framework with Propagation Structure for Boosting Low-Resource Rumor Detection [11.201348902221257]
  existing rumor detection algorithms show promising performance on yesterday's news. Due to a lack of substantial training data and prior expert knowledge, they are poor at spotting rumors concerning unforeseen events. We propose a unified contrastive transfer framework to detect rumors by adapting the features learned from well-resourced rumor data to that of the low-resourced with only few-shot annotations.
  arXiv  Detail & Related papers  (2023-04-04T03:13:03Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Interpretable Fake News Detection with Topic and Deep Variational Models [2.15242029196761]
  We focus on fake news detection using interpretable features and methods. We have developed a deep probabilistic model that integrates a dense representation of textual news. Our model achieves comparable performance to state-of-the-art competing models.
  arXiv  Detail & Related papers  (2022-09-04T05:31:00Z)
• Robust Task-Oriented Dialogue Generation with Contrastive Pre-training and Adversarial Filtering [17.7709632238066]
  Data artifacts incentivize machine learning models to learn non-transferable generalizations. We investigate whether popular datasets such as MultiWOZ contain such data artifacts. We propose a contrastive learning based framework to encourage the model to ignore these cues and focus on learning generalisable patterns.
  arXiv  Detail & Related papers  (2022-05-20T03:13:02Z)
• Detect Rumors in Microblog Posts for Low-Resource Domains via Adversarial Contrastive Learning [8.013665071332388]
  We propose an adversarial contrastive learning framework to detect rumors by adapting the features learned from well-resourced rumor data to that of the low-resourced. Our framework achieves much better performance than state-of-the-art methods and exhibits a superior capacity for detecting rumors at early stages.
  arXiv  Detail & Related papers  (2022-04-18T03:10:34Z)
• Hidden Biases in Unreliable News Detection Datasets [60.71991809782698]
  We show that selection bias during data collection leads to undesired artifacts in the datasets. We observed a significant drop (>10%) in accuracy for all models tested in a clean split with no train/test source overlap. We suggest future dataset creation include a simple model as a difficulty/bias probe and future model development use a clean non-overlapping site and date split.
  arXiv  Detail & Related papers  (2021-04-20T17:16:41Z)
• Learning from Context or Names? An Empirical Study on Neural Relation Extraction [112.06614505580501]
  We study the effect of two main information sources in text: textual context and entity mentions (names) We propose an entity-masked contrastive pre-training framework for relation extraction (RE) Our framework can improve the effectiveness and robustness of neural models in different RE scenarios.
  arXiv  Detail & Related papers  (2020-10-05T11:21:59Z)
• Open-set Short Utterance Forensic Speaker Verification using Teacher-Student Network with Explicit Inductive Bias [59.788358876316295]
  We propose a pipeline solution to improve speaker verification on a small actual forensic field dataset. By leveraging large-scale out-of-domain datasets, a knowledge distillation based objective function is proposed for teacher-student learning. We show that the proposed objective function can efficiently improve the performance of teacher-student learning on short utterances.
  arXiv  Detail & Related papers  (2020-09-21T00:58:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.