Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learning

• URL: http://arxiv.org/abs/2505.06796v1
• Date: Sun, 11 May 2025 00:26:13 GMT
• Title: Multimodal Fake News Detection: MFND Dataset and Shallow-Deep Multitask Learning
• Authors: Ye Zhu, Yunan Wang, Zitong Yu,
• Abstract summary: Multimodal news contains a wealth of information and is easily affected by deepfake modeling attacks.<n>To combat the latest image and text generation methods, we present a new Multimodal Fake News Detection dataset (MFND)<n>MFND contains 11 manipulated types, designed to detect and localize highly authentic fake news.
• Score: 22.494473679788396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal news contains a wealth of information and is easily affected by deepfake modeling attacks. To combat the latest image and text generation methods, we present a new Multimodal Fake News Detection dataset (MFND) containing 11 manipulated types, designed to detect and localize highly authentic fake news. Furthermore, we propose a Shallow-Deep Multitask Learning (SDML) model for fake news, which fully uses unimodal and mutual modal features to mine the intrinsic semantics of news. Under shallow inference, we propose the momentum distillation-based light punishment contrastive learning for fine-grained uniform spatial image and text semantic alignment, and an adaptive cross-modal fusion module to enhance mutual modal features. Under deep inference, we design a two-branch framework to augment the image and text unimodal features, respectively merging with mutual modalities features, for four predictions via dedicated detection and localization projections. Experiments on both mainstream and our proposed datasets demonstrate the superiority of the model. Codes and dataset are released at https://github.com/yunan-wang33/sdml.

Related papers

• KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection [2.3047429933576327]
  We propose a novel multi-modal fake news detection framework that integrates visual, textual, and knowledge-based representations.<n>Our proposal introduces a new paradigm: knowledge-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2025-05-18T13:08:38Z)
• MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching [54.740256498985026]
  Keypoint detection and description methods often struggle with multimodal data.<n>We propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching.
  arXiv  Detail & Related papers  (2025-01-20T06:56:30Z)
• Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation [61.91492500828508]
  Few-shot 3D point cloud segmentation (FS-PCS) aims at generalizing models to segment novel categories with minimal support samples.<n>We introduce a multimodal FS-PCS setup, utilizing textual labels and the potentially available 2D image modality.<n>We propose a simple yet effective Test-time Adaptive Cross-modal (TACC) technique to mitigate training bias.
  arXiv  Detail & Related papers  (2024-10-29T19:28:41Z)
• MMCFND: Multimodal Multilingual Caption-aware Fake News Detection for Low-resource Indic Languages [0.4062349563818079]
  We introduce the Multimodal Multilingual dataset for Indic Fake News Detection (MMIFND) This meticulously curated dataset consists of 28,085 instances distributed across Hindi, Bengali, Marathi, Malayalam, Tamil, Gujarati and Punjabi. We propose the Multimodal Caption-aware framework for Fake News Detection (MMCFND)
  arXiv  Detail & Related papers  (2024-10-14T11:59:33Z)
• Detecting and Grounding Multi-Modal Media Manipulation and Beyond [93.08116982163804]
  We highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM4) DGM4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content. We propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities.
  arXiv  Detail & Related papers  (2023-09-25T15:05:46Z)
• Similarity-Aware Multimodal Prompt Learning for Fake News Detection [0.12396474483677114]
  multimodal fake news detection has outperformed text-only methods. This paper proposes a Similarity-Aware Multimodal Prompt Learning (SAMPLE) framework. For evaluation, SAMPLE surpasses the F1 and the accuracies of previous works on two benchmark multimodal datasets.
  arXiv  Detail & Related papers  (2023-04-09T08:10:05Z)
• Multimodal Fake News Detection with Adaptive Unimodal Representation Aggregation [28.564442206829625]
  AURA is a multimodal fake news detection network with adaptive unimodal representation aggregation. We perform coarse-level fake news detection and cross-modal cosistency learning according to the unimodal and multimodal representations. Experiments on Weibo and Gossipcop prove that AURA can successfully beat several state-of-the-art FND schemes.
  arXiv  Detail & Related papers  (2022-06-12T14:06:55Z)
• Multimodal Fake News Detection via CLIP-Guided Learning [26.093561485807832]
  This paper proposes a FND-CLIP framework, i.e., a multimodal Fake News Detection network based on Contrastive Language-Image Pretraining (CLIP) Given a targeted multimodal news, we extract the deep representations from the image and text using a ResNet-based encoder, a BERT-based encoder and two pair-wise CLIP encoders. The multimodal feature is a concatenation of the CLIP-generated features weighted by the standardized cross-modal similarity of the two modalities.
  arXiv  Detail & Related papers  (2022-05-28T02:43:18Z)
• Weakly Aligned Feature Fusion for Multimodal Object Detection [52.15436349488198]
  multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned. This problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training. In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem.
  arXiv  Detail & Related papers  (2022-04-21T02:35:23Z)
• Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
  We propose to adopt the graph propagation to capture the observed spatial contexts. We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
  arXiv  Detail & Related papers  (2020-08-25T06:00:06Z)
• A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation [131.33610549540043]
  We propose a novel graph-based multi-modal fusion encoder for NMT. We first represent the input sentence and image using a unified multi-modal graph. We then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations.
  arXiv  Detail & Related papers  (2020-07-17T04:06:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.