Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding

• URL: http://arxiv.org/abs/2608.04054v1
• Date: Tue, 04 Aug 2026 10:55:51 GMT
• Title: Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding
• Abstract summary: We propose a hierarchical prototype-hypergraph framework that represents multimodal agreement and conflict as distinct, recurring relational structures.<n>MACH (Modality Agreement- and Conflict-aware prototype Hypergraph) represents multimodal agreement and conflict as distinct, recurring relational structures.
• Score: 2.2012643583422347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal intent recognition requires understanding not only what textual, acoustic, and visual signals share, but also how they disagree. Such disagreement is frequently class-informative; for example, lexical positivity accompanied by incongruent vocal or facial behavior may indicate sarcasm or taunting, yet most fusion methods either encourage modality alignment or treat inconsistency as uncertainty to be suppressed. We propose MACH (Modality Agreement- and Conflict-aware prototype Hypergraph), a hierarchical prototype-hypergraph framework that represents multimodal agreement and conflict as distinct, recurring relational structures. MACH progressively composes unimodal representations into bimodal and trimodal abstractions. At each applicable level, modality-composition anchors activate sparse agreement prototype hypergraphs that capture reusable consensus patterns, while a separate conflict pathway maps cross-modal discrepancies to dedicated conflict prototype hypergraphs. The two pathways are combined through a feature-wise, sample-adaptive arbitration mechanism, enabling the model to preserve informative disagreement while suppressing incidental modality noise. A progressive optimization strategy stabilizes the interdependent hierarchy before joint agreement-conflict learning. Experiments on benchmark datasets demonstrate the effectiveness of the proposed formulation, while component and robustness analyses validate the distinct roles of hierarchical composition, prototype-mediated semantic refinement, and agreement-conflict arbitration.

Related papers

• Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization for Multimodal Sarcasm Detection [8.747820204756566]
  Multimodal sarcasm detection aims to identify sarcastic intent from multimodal content.<n>New framework integrates Dynamic Gated Cross-Modal Fusion with Sarcastic-aware Contrastive Regularization.
  arXiv  Detail & Related papers  (2026-08-20T12:02:06Z)
• CORE: Conflict-Oriented Reasoning for General Multimodal Manipulation Detection [56.64398465636452]
  generative AI has made multimodal fake news increasingly realistic and pervasive, posing severe threats to public trust and social stability.<n>We propose textbfConflict-textbfOriented textbfREasoning (textbfCORE) framework, an effective paradigm that learns to endows multimodal large language models (MLLMs) with explicit conflict-capturing capability.<n> CORE achieves robust and generalizable conflict detection, effectively and rapidly adapting to unseen manipulation types with a few samples or in even zero-shot settings.
  arXiv  Detail & Related papers  (2026-06-02T02:53:48Z)
• To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition [87.22988227382329]
  Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict.<n>We propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities.
  arXiv  Detail & Related papers  (2026-05-06T13:11:33Z)
• PC-MNet: Dual-Level Congruity Modeling for Multimodal Sarcasm Detection via Polarity-Modulated Attention [32.62063298936575]
  This work provides a robust, decoupled paradigm for modeling subtle pragmatic incongruities in human communication.<n>Experiments on the textttMUStARD benchmark and its spurious-correlation-mitigated balanced datasets demonstrate that our approach achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-05-04T10:47:17Z)
• Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning [78.86309644343295]
  Multimodal large language models (MLLMs) in long chain-of-thought reasoning often fail when different knowledge sources provide conflicting signals.<n>We formalize these failures under a unified notion of knowledge conflict, distinguishing input-level objective conflict from process-level effective conflict.<n>Our findings provide a mechanism-level view of multimodal reasoning under knowledge conflict and enable principled diagnosis and control of long-CoT failures.
  arXiv  Detail & Related papers  (2026-02-16T07:10:44Z)
• When Alignment Fails: Multimodal Adversarial Attacks on Vision-Language-Action Models [75.16145284285456]
  We introduce VLA-Fool, a comprehensive study of multimodal adversarial robustness in embodied VLA models under both white-box and black-box settings.<n>We develop the first automatically crafted and semantically guided prompting framework.<n> Experiments on the LIBERO benchmark reveal that even minor multimodal perturbations can cause significant behavioral deviations.
  arXiv  Detail & Related papers  (2025-11-20T10:14:32Z)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning [49.17801010041155]
  Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio.<n>Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance.<n>We categorize multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined.
  arXiv  Detail & Related papers  (2025-09-28T08:46:11Z)
• CLAMP: Contrastive Learning with Adaptive Multi-loss and Progressive Fusion for Multimodal Aspect-Based Sentiment Analysis [0.6961946145048322]
  This paper introduces an end to end Contrastive Learning framework with Adaptive Multi-loss and Progressive Attention Fusion.<n>The framework is composed of three novel modules: Progressive Attention Fusion network, Multi-task Contrastive Learning, and Adaptive Multi-loss Aggregation.<n> evaluation on standard public benchmarks demonstrates that CLAMP consistently outperforms the vast majority of existing state of the art methods.
  arXiv  Detail & Related papers  (2025-07-21T11:49:57Z)
• CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation [24.952907733127223]
  We propose a general framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD)<n>CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates mismatchs while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio)
  arXiv  Detail & Related papers  (2025-05-21T08:11:07Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.