MisRoB{\AE}RTa: Transformers versus Misinformation

• URL: http://arxiv.org/abs/2304.07759v1
• Date: Sun, 16 Apr 2023 12:14:38 GMT
• Title: MisRoB{\AE}RTa: Transformers versus Misinformation
• Authors: Ciprian-Octavian Truic\u{a} and Elena-Simona Apostol
• Abstract summary: We propose a novel transformer-based deep neural ensemble architecture for misinformation detection. MisRoBAERTa takes advantage of two transformers (BART & RoBERTa) to improve the classification performance. For training and testing, we used a large real-world news articles dataset labeled with 10 classes.
• Score: 0.6091702876917281
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Misinformation is considered a threat to our democratic values and principles. The spread of such content on social media polarizes society and undermines public discourse by distorting public perceptions and generating social unrest while lacking the rigor of traditional journalism. Transformers and transfer learning proved to be state-of-the-art methods for multiple well-known natural language processing tasks. In this paper, we propose MisRoB{\AE}RTa, a novel transformer-based deep neural ensemble architecture for misinformation detection. MisRoB{\AE}RTa takes advantage of two transformers (BART \& RoBERTa) to improve the classification performance. We also benchmarked and evaluated the performances of multiple transformers on the task of misinformation detection. For training and testing, we used a large real-world news articles dataset labeled with 10 classes, addressing two shortcomings in the current research: increasing the size of the dataset from small to large, and moving the focus of fake news detection from binary classification to multi-class classification. For this dataset, we manually verified the content of the news articles to ensure that they were correctly labeled. The experimental results show that the accuracy of transformers on the misinformation detection problem was significantly influenced by the method employed to learn the context, dataset size, and vocabulary dimension. We observe empirically that the best accuracy performance among the classification models that use only one transformer is obtained by BART, while DistilRoBERTa obtains the best accuracy in the least amount of time required for fine-tuning and training. The proposed MisRoB{\AE}RTa outperforms the other transformer models in the task of misinformation detection. To arrive at this conclusion, we performed ample ablation and sensitivity testing with MisRoB{\AE}RTa on two datasets.

Related papers

• Identifying Misinformation on YouTube through Transcript Contextual Analysis with Transformer Models [1.749935196721634]
  We introduce a novel methodology for video classification, focusing on the veracity of the content. We employ advanced machine learning techniques like transfer learning to solve the classification challenge. We apply the trained models to three datasets: (a) YouTube Vaccine-misinformation related videos, (b) YouTube Pseudoscience videos, and (c) Fake-News dataset.
  arXiv  Detail & Related papers  (2023-07-22T19:59:16Z)
• Remote Sensing Change Detection With Transformers Trained from Scratch [62.96911491252686]
  transformer-based change detection (CD) approaches either employ a pre-trained model trained on large-scale image classification ImageNet dataset or rely on first pre-training on another CD dataset and then fine-tuning on the target benchmark. We develop an end-to-end CD approach with transformers that is trained from scratch and yet achieves state-of-the-art performance on four public benchmarks.
  arXiv  Detail & Related papers  (2023-04-13T17:57:54Z)
• Transformer-based approaches to Sentiment Detection [55.41644538483948]
  We examined the performance of four different types of state-of-the-art transformer models for text classification. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions.
  arXiv  Detail & Related papers  (2023-03-13T17:12:03Z)
• Mitigating Representation Bias in Action Recognition: Algorithms and Benchmarks [76.35271072704384]
  Deep learning models perform poorly when applied to videos with rare scenes or objects. We tackle this problem from two different angles: algorithm and dataset. We show that the debiased representation can generalize better when transferred to other datasets and tasks.
  arXiv  Detail & Related papers  (2022-09-20T00:30:35Z)
• Detecting COVID-19 Conspiracy Theories with Transformers and TF-IDF [2.3202611780303553]
  We present our methods and results for three fake news detection tasks at MediaEval benchmark 2021. We find that a pre-trained transformer yields the best validation results, but a randomly trained transformer with smart design can also be trained to reach accuracies close to that of the pre-trained transformer.
  arXiv  Detail & Related papers  (2022-05-01T01:48:48Z)
• Towards Data-Efficient Detection Transformers [77.43470797296906]
  We show most detection transformers suffer from significant performance drops on small-size datasets. We empirically analyze the factors that affect data efficiency, through a step-by-step transition from a data-efficient RCNN variant to the representative DETR. We introduce a simple yet effective label augmentation method to provide richer supervision and improve data efficiency.
  arXiv  Detail & Related papers  (2022-03-17T17:56:34Z)
• Transforming Fake News: Robust Generalisable News Classification Using Transformers [8.147652597876862]
  Using the publicly available ISOT and Combined Corpus datasets, this study explores transformers' abilities to identify fake news. We propose a novel two-step classification pipeline to remove such articles from both model training and the final deployed inference system. Experiments over the ISOT and Combined Corpus datasets show that transformers achieve an increase in F1 scores of up to 4.9% for out of distribution generalisation.
  arXiv  Detail & Related papers  (2021-09-20T19:03:16Z)
• Understanding Transformers for Bot Detection in Twitter [0.0]
  We focus on bot detection in Twitter, a key task to mitigate and counteract the automatic spreading of disinformation and bias in social media. We investigate the use of pre-trained language models to tackle the detection of tweets generated by a bot or a human account based exclusively on its content. We observe that fine-tuning generative transformers on a bot detection task produces higher accuracies.
  arXiv  Detail & Related papers  (2021-04-13T13:32:55Z)
• Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection [7.29381091750894]
  We propose a novel transformer-based language model fine-tuning approach for these fake news detection. First, the token vocabulary of individual model is expanded for the actual semantics of professional phrases. Last, the predicted features extracted by universal language model RoBERTa and domain-specific model CT-BERT are fused by one multiple layer perception to integrate fine-grained and high-level specific representations.
  arXiv  Detail & Related papers  (2021-01-14T09:05:42Z)
• Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks [75.69896269357005]
  Mixup is the latest data augmentation technique that linearly interpolates input examples and the corresponding labels. In this paper, we explore how to apply mixup to natural language processing tasks. We incorporate mixup to transformer-based pre-trained architecture, named "mixup-transformer", for a wide range of NLP tasks.
  arXiv  Detail & Related papers  (2020-10-05T23:37:30Z)
• The Cascade Transformer: an Application for Efficient Answer Sentence Selection [116.09532365093659]
  We introduce the Cascade Transformer, a technique to adapt transformer-based models into a cascade of rankers. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy.
  arXiv  Detail & Related papers  (2020-05-05T23:32:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.