Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

• URL: http://arxiv.org/abs/2603.26052v1
• Date: Fri, 27 Mar 2026 03:38:38 GMT
• Title: Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification
• Abstract summary: MaLSF (Mask-aware Local Semantic Fusion) is a novel framework that shifts the paradigm to active, bidirectional verification.<n>It uses mask-label pairs as semantic anchors to bridge pixels and words.<n>MaLSF achieves state-of-the-art performance on both the DGM4 and multimodal fake news detection tasks.
• Score: 13.571218577944032
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As multimodal misinformation becomes more sophisticated, its detection and grounding are crucial. However, current multimodal verification methods, relying on passive holistic fusion, struggle with sophisticated misinformation. Due to 'feature dilution,' global alignments tend to average out subtle local semantic inconsistencies, effectively masking the very conflicts they are designed to find. We introduce MaLSF (Mask-aware Local Semantic Fusion), a novel framework that shifts the paradigm to active, bidirectional verification, mimicking human cognitive cross-referencing. MaLSF utilizes mask-label pairs as semantic anchors to bridge pixels and words. Its core mechanism features two innovations: 1) a Bidirectional Cross-modal Verification (BCV) module that acts as an interrogator, using parallel query streams (Text-as-Query and Image-as-Query) to explicitly pinpoint conflicts; and 2) a Hierarchical Semantic Aggregation (HSA) module that intelligently aggregates these multi-granularity conflict signals for task-specific reasoning. In addition, to extract fine-grained mask-label pairs, we introduce a set of diverse mask-label pair extraction parsers. MaLSF achieves state-of-the-art performance on both the DGM4 and multimodal fake news detection tasks. Extensive ablation studies and visualization results further verify its effectiveness and interpretability.

Related papers

• Multi-Modal Object Re-Identification with Dual Semantic Guidance and Global-Local Mutual Modulation [43.18497349594355]
  Multi-modal object Re-Identification (ReID) aims to retrieve target instances by leveraging complementary information across modalities.<n>Existing methods fail to exploit well-aligned and reliable semantic priors, making them vulnerable to background clutter and cross-modal misalignment.<n>We propose a robust multi-modal ReID framework with dual semantic guidance and global-local mutual modulation.
  arXiv  Detail & Related papers  (2026-07-31T09:27:10Z)
• SAC$^2$-Net: Semantic Anchoring and Complementary-Consensus Fusion for Multimodal Micro-Expression Recognition [5.567888983005847]
  Micro-expression recognition (MER) is challenging due to subtle facial movements, limited data, and the ambiguous relationship between Action Units (AUs) and emotion categories.<n>We propose SAC$2$-Net, a Semantic Anchoring and Complementary-Consensus Network for multimodal MER.<n>We show that SAC$2$-Net achieves state-of-the-art or highly competitive performance across coarse-grained, fine-grained, large-scale, and cross-dataset evaluation settings.
  arXiv  Detail & Related papers  (2026-06-24T08:20:03Z)
• SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignment [8.657941729790599]
  We introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity.<n>Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches.<n>Experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance.
  arXiv  Detail & Related papers  (2025-11-03T09:41:32Z)
• Beyond Artificial Misalignment: Detecting and Grounding Semantic-Coordinated Multimodal Manipulations [56.816929931908824]
  We pioneer the detection of semantically-coordinated manipulations in multimodal data.<n>We propose a Retrieval-Augmented Manipulation Detection and Grounding (RamDG) framework.<n>Our framework significantly outperforms existing methods, achieving 2.06% higher detection accuracy on SAMM compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-16T04:18:48Z)
• KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection [3.221633413708485]
  We propose a novel multi-modal fake news detection framework that integrates visual, textual, and knowledge-based representations.<n>Our proposal introduces a new paradigm: knowledge-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2025-05-18T13:08:38Z)
• Robust Multi-View Learning via Representation Fusion of Sample-Level Attention and Alignment of Simulated Perturbation [61.64052577026623]
  Real-world multi-view datasets are often heterogeneous and imperfect.<n>We propose a novel robust MVL method (namely RML) with simultaneous representation fusion and alignment.<n>Our RML is self-supervised and can also be applied for downstream tasks as a regularization.
  arXiv  Detail & Related papers  (2025-03-06T07:01:08Z)
• Mixture-of-Noises Enhanced Forgery-Aware Predictor for Multi-Face Manipulation Detection and Localization [52.87635234206178]
  This paper proposes a new framework, namely MoNFAP, specifically tailored for multi-face manipulation detection and localization. The framework incorporates two novel modules: the Forgery-aware Unified Predictor (FUP) Module and the Mixture-of-Noises Module (MNM)
  arXiv  Detail & Related papers  (2024-08-05T08:35:59Z)
• Detecting and Grounding Multi-Modal Media Manipulation and Beyond [93.08116982163804]
  We highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM4) DGM4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content. We propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities.
  arXiv  Detail & Related papers  (2023-09-25T15:05:46Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)
• Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition [63.07844685982738]
  This paper presents a new model named as Gated Bidirectional Alignment Network (GBAN), which consists of an attention-based bidirectional alignment network over LSTM hidden states. We empirically show that the attention-aligned representations outperform the last-hidden-states of LSTM significantly. The proposed GBAN model outperforms existing state-of-the-art multimodal approaches on the IEMOCAP dataset.
  arXiv  Detail & Related papers  (2022-01-17T09:46:59Z)
• CMF: Cascaded Multi-model Fusion for Referring Image Segmentation [24.942658173937563]
  We address the task of referring image segmentation (RIS), which aims at predicting a segmentation mask for the object described by a natural language expression. We propose a simple yet effective Cascaded Multi-modal Fusion (CMF) module, which stacks multiple atrous convolutional layers in parallel. Experimental results on four benchmark datasets demonstrate that our method outperforms most state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-16T08:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.