Fine-grained Multiple Supervisory Network for Multi-modal Manipulation Detecting and Grounding

• URL: http://arxiv.org/abs/2508.02479v1
• Date: Mon, 04 Aug 2025 14:46:59 GMT
• Title: Fine-grained Multiple Supervisory Network for Multi-modal Manipulation Detecting and Grounding
• Authors: Xinquan Yu, Wei Lu, Xiangyang Luo,
• Abstract summary: Multi-Modal Media Manipulation (DGM$4$) is a branch of misinformation detection.<n>Existing methods are often limited in performance due to the erroneous interference caused by unreliable unimodal data.<n>We present a Fine-grained Multiple Supervisory (FMS) network, which incorporates modality reliability supervision, unimodal internal supervision and cross-modal supervision.
• Score: 15.670987587762259
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The task of Detecting and Grounding Multi-Modal Media Manipulation (DGM$^4$) is a branch of misinformation detection. Unlike traditional binary classification, it includes complex subtasks such as forgery content localization and forgery method classification. Consider that existing methods are often limited in performance due to neglecting the erroneous interference caused by unreliable unimodal data and failing to establish comprehensive forgery supervision for mining fine-grained tampering traces. In this paper, we present a Fine-grained Multiple Supervisory (FMS) network, which incorporates modality reliability supervision, unimodal internal supervision and cross-modal supervision to provide comprehensive guidance for DGM$^4$ detection. For modality reliability supervision, we propose the Multimodal Decision Supervised Correction (MDSC) module. It leverages unimodal weak supervision to correct the multi-modal decision-making process. For unimodal internal supervision, we propose the Unimodal Forgery Mining Reinforcement (UFMR) module. It amplifies the disparity between real and fake information within unimodal modality from both feature-level and sample-level perspectives. For cross-modal supervision, we propose the Multimodal Forgery Alignment Reasoning (MFAR) module. It utilizes soft-attention interactions to achieve cross-modal feature perception from both consistency and inconsistency perspectives, where we also design the interaction constraints to ensure the interaction quality. Extensive experiments demonstrate the superior performance of our FMS compared to state-of-the-art methods.

Related papers

• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [50.438552588818]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• Evaluating and Steering Modality Preferences in Multimodal Large Language Model [32.94581875014947]
  Multimodal large language models (MLLMs) have achieved remarkable performance on complex tasks with multimodal context.<n>We show that all 18 tested MLLMs generally demonstrate clear modality bias, and modality preference can be influenced by external interventions.<n>We propose a probing and steering method based on representation engineering to explicitly control modality preference.
  arXiv  Detail & Related papers  (2025-05-27T10:07:59Z)
• Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models [26.17300490736624]
  Multimodal Large Language Models (MLLMs) are predominantly trained and tested on consistent visual-textual inputs.<n>We propose the Multimodal Inconsistency Reasoning benchmark to assess MLLMs' ability to detect and reason about semantic mismatches.<n>We evaluate six state-of-the-art MLLMs, showing that models with dedicated multimodal reasoning capabilities, such as o1, substantially outperform their counterparts.
  arXiv  Detail & Related papers  (2025-02-22T01:52:37Z)
• Multimodal Task Representation Memory Bank vs. Catastrophic Forgetting in Anomaly Detection [6.991692485111346]
  Unsupervised Continuous Anomaly Detection (UCAD) faces significant challenges in multi-task representation learning.<n>We propose the Multimodal Task Representation Memory Bank (MTRMB) method through two key technical innovations.<n> Experiments on MVtec AD and VisA datasets demonstrate MTRMB's superiority, achieving an average detection accuracy of 0.921 at the lowest forgetting rate.
  arXiv  Detail & Related papers  (2025-02-10T06:49:54Z)
• Tuned Reverse Distillation: Enhancing Multimodal Industrial Anomaly Detection with Crossmodal Tuners [15.89869857998053]
  We propose Tuned Reverse Distillation (TRD) based on Multi-branch design to realize Multimodal Industrial AD.<n>By assigning independent branches to each modality, our method enables finer detection of anomalies within each modality.<n>Our method achieves state-of-the-art performance in multimodal anomaly detection and localization.
  arXiv  Detail & Related papers  (2024-12-12T05:26:50Z)
• RADAR: Robust Two-stage Modality-incomplete Industrial Anomaly Detection [61.71770293720491]
  We propose a novel two-stage Robust modAlity-imcomplete fusing and Detecting frAmewoRk, abbreviated as RADAR. Our bootstrapping philosophy is to enhance two stages in MIIAD, improving the robustness of the Multimodal Transformer. Our experimental results demonstrate that the proposed RADAR significantly surpasses conventional MIAD methods in terms of effectiveness and robustness.
  arXiv  Detail & Related papers  (2024-10-02T16:47:55Z)
• DMM: Disparity-guided Multispectral Mamba for Oriented Object Detection in Remote Sensing [8.530409994516619]
  Multispectral oriented object detection faces challenges due to both inter-modal and intra-modal discrepancies. We propose Disparity-guided Multispectral Mamba (DMM), a framework comprised of a Disparity-guided Cross-modal Fusion Mamba (DCFM) module, a Multi-scale Target-aware Attention (MTA) module, and a Target-Prior Aware (TPA) auxiliary task.
  arXiv  Detail & Related papers  (2024-07-11T02:09:59Z)
• Detecting and Grounding Multi-Modal Media Manipulation and Beyond [93.08116982163804]
  We highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM4) DGM4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content. We propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities.
  arXiv  Detail & Related papers  (2023-09-25T15:05:46Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis [18.833050804875032]
  Multimodal sentiment analysis (MSA) draws increasing attention with the availability of multimodal data. Recent MSA works mostly focus on learning cross-modal dynamics, but neglect to explore an optimal solution for unimodal networks. We propose a novel MSA framework textbfModulation textbfModel for textbfMultimodal textbfSentiment textbfAnalysis.
  arXiv  Detail & Related papers  (2021-11-10T03:29:17Z)
• Digging into Uncertainty in Self-supervised Multi-view Stereo [57.04768354383339]
  We propose a novel Uncertainty reduction Multi-view Stereo (UMVS) framework for self-supervised learning. Our framework achieves the best performance among unsupervised MVS methods, with competitive performance with its supervised opponents.
  arXiv  Detail & Related papers  (2021-08-30T02:53:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.