Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation

• URL: http://arxiv.org/abs/2112.08594v1
• Date: Thu, 16 Dec 2021 03:37:20 GMT
• Title: Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation
• Authors: Giscard Biamby, Grace Luo, Trevor Darrell, Anna Rohrbach
• Abstract summary: This paper describes our approach to the Image-Text Inconsistency Detection challenge of the DARPA Semantic Forensics (SemaFor) Program. We collect Twitter-COMMs, a large-scale multimodal dataset with 884k tweets relevant to the topics of Climate Change, COVID-19, and Military Vehicles. We train our approach, based on the state-of-the-art CLIP model, leveraging automatically generated random and hard negatives.
• Score: 83.2079454464572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Detecting out-of-context media, such as "miscaptioned" images on Twitter, often requires detecting inconsistencies between the two modalities. This paper describes our approach to the Image-Text Inconsistency Detection challenge of the DARPA Semantic Forensics (SemaFor) Program. First, we collect Twitter-COMMs, a large-scale multimodal dataset with 884k tweets relevant to the topics of Climate Change, COVID-19, and Military Vehicles. We train our approach, based on the state-of-the-art CLIP model, leveraging automatically generated random and hard negatives. Our method is then tested on a hidden human-generated evaluation set. We achieve the best result on the program leaderboard, with 11% detection improvement in a high precision regime over a zero-shot CLIP baseline.

Related papers

• Fraunhofer SIT at CheckThat! 2023: Mixing Single-Modal Classifiers to Estimate the Check-Worthiness of Multi-Modal Tweets [0.0]
  This paper proposes a novel way of detecting the check-worthiness in multi-modal tweets. It takes advantage of two classifiers, each trained on a single modality. For image data, extracting the embedded text with an OCR analysis has shown to perform best.
  arXiv  Detail & Related papers  (2023-07-02T16:35:54Z)
• It's Just a Matter of Time: Detecting Depression with Time-Enriched Multimodal Transformers [24.776445591293186]
  We propose a flexible time-enriched multimodal transformer architecture for detecting depression from social media posts. Our model operates directly at the user-level, and we enrich it with the relative time between posts by using time2vec positional embeddings. We show that our method, using EmoBERTa and CLIP embeddings, surpasses other methods on two multimodal datasets.
  arXiv  Detail & Related papers  (2023-01-13T09:40:19Z)
• Machine Learning-based Automatic Annotation and Detection of COVID-19 Fake News [8.020736472947581]
  COVID-19 impacted every part of the world, although the misinformation about the outbreak traveled faster than the virus. Existing work neglects the presence of bots that act as a catalyst in the spread. We propose an automated approach for labeling data using verified fact-checked statements on a Twitter dataset.
  arXiv  Detail & Related papers  (2022-09-07T13:55:59Z)
• Identification of Twitter Bots based on an Explainable ML Framework: the US 2020 Elections Case Study [72.61531092316092]
  This paper focuses on the design of a novel system for identifying Twitter bots based on labeled Twitter data. Supervised machine learning (ML) framework is adopted using an Extreme Gradient Boosting (XGBoost) algorithm. Our study also deploys Shapley Additive Explanations (SHAP) for explaining the ML model predictions.
  arXiv  Detail & Related papers  (2021-12-08T14:12:24Z)
• SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving [94.11868795445798]
  We release a Large-Scale Object Detection benchmark for Autonomous driving, named as SODA10M, containing 10 million unlabeled images and 20K images labeled with 6 representative object categories. To improve diversity, the images are collected every ten seconds per frame within 32 different cities under different weather conditions, periods and location scenes. We provide extensive experiments and deep analyses of existing supervised state-of-the-art detection models, popular self-supervised and semi-supervised approaches, and some insights about how to develop future models.
  arXiv  Detail & Related papers  (2021-06-21T13:55:57Z)
• Holistic Image Manipulation Detection using Pixel Co-occurrence Matrices [16.224649756613655]
  Digital image forensics aims to detect images that have been digitally manipulated. Most detection methods in literature focus on detecting a particular type of manipulation. We propose a novel approach to holistically detect tampered images using a combination of pixel co-occurrence matrices and deep learning.
  arXiv  Detail & Related papers  (2021-04-12T17:54:42Z)
• Combining exogenous and endogenous signals with a semi-supervised co-attention network for early detection of COVID-19 fake tweets [14.771202995527315]
  During COVID-19, tweets with misinformation should be flagged and neutralized in their early stages to mitigate the damages. Most of the existing methods for early detection of fake news assume to have enough propagation information for large labeled tweets. We present ENDEMIC, a novel early detection model which leverages endogenous and endogenous signals related to tweets.
  arXiv  Detail & Related papers  (2021-04-12T10:01:44Z)
• Quasi-Dense Similarity Learning for Multiple Object Tracking [82.93471035675299]
  We present Quasi-Dense Similarity Learning, which densely samples hundreds of region proposals on a pair of images for contrastive learning. We can directly combine this similarity learning with existing detection methods to build Quasi-Dense Tracking (QDTrack)
  arXiv  Detail & Related papers  (2020-06-11T17:57:12Z)
• SCAN: Learning to Classify Images without Labels [73.69513783788622]
  We advocate a two-step approach where feature learning and clustering are decoupled. A self-supervised task from representation learning is employed to obtain semantically meaningful features. We obtain promising results on ImageNet, and outperform several semi-supervised learning methods in the low-data regime.
  arXiv  Detail & Related papers  (2020-05-25T18:12:33Z)
• Tracking Road Users using Constraint Programming [79.32806233778511]
  We present a constraint programming (CP) approach for the data association phase found in the tracking-by-detection paradigm of the multiple object tracking (MOT) problem. Our proposed method was tested on a motorized vehicles tracking dataset and produces results that outperform the top methods of the UA-DETRAC benchmark.
  arXiv  Detail & Related papers  (2020-03-10T00:04:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.