Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection

• URL: http://arxiv.org/abs/2608.19942v1
• Date: Thu, 20 Aug 2026 12:02:06 GMT
• Title: Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection
• Abstract summary: Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content.<n>New framework integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization.
• Score: 8.747820204756566
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content, where inconsistencies between literal meaning and contextual cues often signal irony. This task has attracted increasing research attention. However, accurate detection remains challenging due to instance-dependent modality contributions and misleading semantic consistency, where surface-level alignment masks underlying contradictory intent. Existing methods often rely on fixed fusion strategies and treat sarcasm as generic cross-modal mismatch, limiting their ability to capture subtle sarcasm cues and instance-specific modality interactions. To address these challenges, we propose a novel MSD framework that integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization (SaCR). Specifically, a bidirectional gated interaction module performs cross-modal feature filtering and adaptively calibrates textual and visual contributions at the instance level. A dynamic fusion gate further balances modality importance to generate more robust multimodal representations. Furthermore, SaCR is introduced as a label-aware contrastive regularization objective that encourages semantic consistency for non-sarcastic samples while suppressing misleading consistency in sarcastic cases. The proposed framework is trained end-to-end with a multi-objective learning strategy that jointly optimizes multimodal classification and auxiliary unimodal supervision. Extensive experiments on MMSD and MMSD2.0 demonstrate that the proposed method consistently outperforms strong baselines.

Related papers

• AdaSurvMamba: Dynamic Fusion and Semantic Scanning for Multimodal Survival Analysis [57.942670844512016]
  We introduce AdaSurvMamba as a novel adaptive framework for multimodal survival analysis.<n>The framework features a Dual-Scale Importance-Aware Reconstruction (DSIR) module to dynamically modulate cross-modal interaction strength.<n>Experiments across five TCGA cohorts demonstrate consistent gains over existing methods.
  arXiv  Detail & Related papers  (2026-06-28T11:36:31Z)
• PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention [32.62063298936575]
  This work provides a robust, decoupled paradigm for modeling subtle pragmatic incongruities in human communication.<n>Experiments on the textttMUStARD benchmark and its spurious-correlation-mitigated balanced datasets demonstrate that our approach achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-05-04T10:47:17Z)
• Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoning [65.15766304205657]
  In-context learning (ICL) allows large models to adapt to tasks using a few examples, yet its extension to vision-language models (VLMs) remains fragile.<n>Our analysis reveals that the fundamental limitation lies in an inductive gap, models often produce correct answers from flawed reasoning.<n>We introduce a framework that restructures multimodal ICL as a principled inductive-deductive process.
  arXiv  Detail & Related papers  (2026-05-04T09:18:19Z)
• URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection [17.19666367498091]
  Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image.<n>We propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during interaction and fusion.<n>URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines.
  arXiv  Detail & Related papers  (2026-04-08T06:50:43Z)
• SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data [31.146366498415784]
  Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation.<n>IMSS faces three key challenges: multimodal imbalance, where dominant modalities suppress fragile ones; intra-class variation in scale, shape, and orientation across modalities; and cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses.<n>We propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance.
  arXiv  Detail & Related papers  (2026-03-03T01:28:21Z)
• MuVaC: AVariational Causal Framework for Multimodal Sarcasm Understanding in Dialogues [21.146757458620105]
  Sarcasm analysis requires Multimodal Sarcasm Detection (MSD) and Multimodal Sarcasm Explanation (MuSE)<n>We propose MuVaC, a variational causal inference framework that mimics human cognitive mechanisms for understanding sarcasm.
  arXiv  Detail & Related papers  (2026-01-28T10:19:42Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• DADM: Dual Alignment of Domain and Modality for Face Anti-spoofing [58.62312400472865]
  Multi-modal face anti-spoofing (FAS) has emerged as a prominent research focus.<n>We propose a alignment module between modalities based on mutual information.<n>We employ a dual alignment optimization method that aligns both sub-domain hyperplanes and modality angle margins.
  arXiv  Detail & Related papers  (2025-03-01T10:12:00Z)
• Multi-View Incongruity Learning for Multimodal Sarcasm Detection [40.10921890527881]
  Multimodal sarcasm detection (MSD) is essential for various downstream tasks.<n>Existing MSD methods tend to rely on spurious correlations.<n>This paper proposes a novel method that integrate Multimodal Incongruities via Contrastive Learning (MICL) for multimodal sarcasm detection.
  arXiv  Detail & Related papers  (2024-12-01T10:29:36Z)
• Modality Prompts for Arbitrary Modality Salient Object Detection [57.610000247519196]
  This paper delves into the task of arbitrary modality salient object detection (AM SOD) It aims to detect salient objects from arbitrary modalities, eg RGB images, RGB-D images, and RGB-D-T images. A novel modality-adaptive Transformer (MAT) will be proposed to investigate two fundamental challenges of AM SOD.
  arXiv  Detail & Related papers  (2024-05-06T11:02:02Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)
• Object Segmentation by Mining Cross-Modal Semantics [68.88086621181628]
  We propose a novel approach by mining the Cross-Modal Semantics to guide the fusion and decoding of multimodal features. Specifically, we propose a novel network, termed XMSNet, consisting of (1) all-round attentive fusion (AF), (2) coarse-to-fine decoder (CFD), and (3) cross-layer self-supervision.
  arXiv  Detail & Related papers  (2023-05-17T14:30:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.