Multimodal Misinformation Detection Using Early Fusion of Linguistic, Visual, and Social Features

• URL: http://arxiv.org/abs/2507.01984v1
• Date: Thu, 26 Jun 2025 18:17:35 GMT
• Title: Multimodal Misinformation Detection Using Early Fusion of Linguistic, Visual, and Social Features
• Authors: Gautam Kishore Shahi,
• Abstract summary: This study analyzed 1,529 tweets containing both text and images during the COVID-19 pandemic and election periods collected from Twitter (now X)<n>The results show that combining unsupervised and supervised machine learning models improves classification performance by 15% compared to unimodal models and by 5% compared to bimodal models.
• Score: 6.8894258727040665
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Amid a tidal wave of misinformation flooding social media during elections and crises, extensive research has been conducted on misinformation detection, primarily focusing on text-based or image-based approaches. However, only a few studies have explored multimodal feature combinations, such as integrating text and images for building a classification model to detect misinformation. This study investigates the effectiveness of different multimodal feature combinations, incorporating text, images, and social features using an early fusion approach for the classification model. This study analyzed 1,529 tweets containing both text and images during the COVID-19 pandemic and election periods collected from Twitter (now X). A data enrichment process was applied to extract additional social features, as well as visual features, through techniques such as object detection and optical character recognition (OCR). The results show that combining unsupervised and supervised machine learning models improves classification performance by 15% compared to unimodal models and by 5% compared to bimodal models. Additionally, the study analyzes the propagation patterns of misinformation based on the characteristics of misinformation tweets and the users who disseminate them.

Related papers

• CLIP-IT: CLIP-based Pairing for Histology Images Classification [6.855390956571216]
  We introduce CLIP-IT to train a vision backbone model to classify histology images by pairing them with privileged textual information from an external source.<n>At first, the modality pairing step relies on a CLIP-based model to match histology images with semantically relevant textual report data from external sources, creating an augmented multimodal dataset.<n>A parameter-efficient fine-tuning method is used to efficiently address the misalignment between the main (image) and paired (text) modalities.
  arXiv  Detail & Related papers  (2025-04-22T18:14:43Z)
• A New Hybrid Intelligent Approach for Multimodal Detection of Suspected Disinformation on TikTok [0.0]
  This study introduces a hybrid framework that combines the computational power of deep learning with the interpretability of fuzzy logic to detect suspected disinformation in TikTok videos.<n>The methodology is comprised of two core components: a multimodal feature analyser that extracts and evaluates data from text, audio, and video; and a multimodal disinformation detector based on fuzzy logic.
  arXiv  Detail & Related papers  (2025-02-09T12:37:48Z)
• A Self-Learning Multimodal Approach for Fake News Detection [35.98977478616019]
  We introduce a self-learning multimodal model for fake news classification.<n>The model leverages contrastive learning, a robust method for feature extraction that operates without requiring labeled data.<n>Our experimental results on a public dataset demonstrate that the proposed model outperforms several state-of-the-art classification approaches.
  arXiv  Detail & Related papers  (2024-12-08T07:41:44Z)
• Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning [7.84845040922464]
  We present our solution for SMART-101 Challenge of CVPR Multi-modal Algorithmic Reasoning Task 2024. Unlike traditional visual questions and answer tasks, this challenge evaluates abstraction, deduction and generalization ability of neural network. Our model is based on two pre-trained models, dedicated to extract features from text and image respectively.
  arXiv  Detail & Related papers  (2024-06-08T01:45:06Z)
• Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers [120.49126407479717]
  This paper explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR) We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos.
  arXiv  Detail & Related papers  (2024-03-12T00:02:03Z)
• On the Multi-modal Vulnerability of Diffusion Models [56.08923332178462]
  We propose MMP-Attack to manipulate the generation results of diffusion models by appending a specific suffix to the original prompt.<n>Our goal is to induce diffusion models to generate a specific object while simultaneously eliminating the original object.
  arXiv  Detail & Related papers  (2024-02-02T12:39:49Z)
• Parkinson's Disease Classification Using Contrastive Graph Cross-View Learning with Multimodal Fusion of SPECT Images and Clinical Features [5.660131312162423]
  Parkinson's Disease (PD) affects millions globally, impacting movement. Prior research utilized deep learning for PD prediction, primarily focusing on medical images, neglecting the data's underlying manifold structure. This work proposes a multimodal approach encompassing both image and non-image features, leveraging contrastive cross-view graph fusion for PD classification.
  arXiv  Detail & Related papers  (2023-11-25T02:32:46Z)
• Two Approaches to Supervised Image Segmentation [55.616364225463066]
  The present work develops comparison experiments between deep learning and multiset neurons approaches. The deep learning approach confirmed its potential for performing image segmentation. The alternative multiset methodology allowed for enhanced accuracy while requiring little computational resources.
  arXiv  Detail & Related papers  (2023-07-19T16:42:52Z)
• UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning [86.91893533388628]
  This paper presents UniDiff, a unified multi-modal model that integrates image-text contrastive learning (ITC), text-conditioned image synthesis learning (IS), and reciprocal semantic consistency modeling (RSC) UniDiff demonstrates versatility in both multi-modal understanding and generative tasks.
  arXiv  Detail & Related papers  (2023-06-01T15:39:38Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal Classification [5.960550152906609]
  We capture hinting features from user comments, which are retrieved via jointly leveraging visual and lingual similarity. The classification tasks are explored via self-training in a teacher-student framework, motivated by the usually limited labeled data scales. The results show that our method further advances the performance of previous state-of-the-art models.
  arXiv  Detail & Related papers  (2023-03-27T08:59:55Z)
• Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings [63.79979145520512]
  We explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. We propose a novel Multi-Modality Multi-Head Attention (M3H-Att) to capture the intricate cross-media interactions. Our model significantly outperforms the previous state of the art based on traditional attention networks.
  arXiv  Detail & Related papers  (2020-11-03T08:44:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.