To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition

• URL: http://arxiv.org/abs/2605.04877v1
• Date: Wed, 06 May 2026 13:11:33 GMT
• Title: To Fuse or to Drop? Dual-Path Learning for Resolving Modality Conflicts in Multimodal Emotion Recognition
• Abstract summary: Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict.<n>We propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities.
• Score: 87.22988227382329
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal emotion recognition (MER) benefits from combining text, audio, and vision, yet standard fusion often fails when modalities conflict. Crucially, conflicts differ in resolvability: benign conflicts stem from missing, weak, or ambiguous cues and can be mitigated by cross-modal calibration, while severe conflicts arise from intrinsically contradictory (e.g., sarcasm) or misleading signals, for which forced fusion may amplify errors. Recognizing this, we propose Dual-Path Conflict Resolution (DCR), a unified framework that learns when to fuse and when to drop modalities. Path I (Affective Fusion Distiller, AFD) performs reverse distillation from audio/visual teachers to a textual student using temporally weighted class evidence, thereby enhancing representation-level calibration and improving fusion when alignment is beneficial. Path II (Affective Discernment Agent, ADA) formulates MER as a contextual bandit that selects among fusion and unimodal predictions based on a dual-view state and a calibration-aware reward, enabling decision-level arbitration under irreconcilable conflicts without requiring per-modality reliability labels. By taking into account the full multimodal context and coupling soft calibration with hard arbitration, DCR reconciles conflicts that can be aligned while bypassing misleading modalities when fusion is harmful. Across five benchmarks covering both dialogue-level and clip-level MER, DCR consistently outperforms competitive baselines or achieves highly competitive results. Further ablations, conflict-specific subset evaluation, and modality-selection analysis verify that AFD and ADA are complementary and jointly improve robust conflict-aware emotion recognition.

Related papers

• CONFER: Conflict-Aware Evidence Negotiation for Regime-Calibrated Weak Supervision in Multimodal Emotion Recognition [2.587219755021037]
  graph-based conflict-aware evidence negotiation framework for weakly supervised multimodal emotion recognition.<n>ConFER represents each modality expert as a node with a predictive belief, boundary-based uncertainty, and runtime reliability estimated from historical out-of-fold performance and current-sample uncertainty.<n>ConFER achieves competitive performance, reaching textbf0.873 accuracy on AMIGOS-V and textbf0.854 accuracy on MAHNOB-V under strict LOSO evaluation.
  arXiv  Detail & Related papers  (2026-08-08T02:42:00Z)
• CARGO-VL: Counterfactual Arbitration with Risk-Constrained Group Optimization for Vision-Language Models [2.038560872025072]
  Vision-language systems combine images with retrieved text, but these sources can disagree or jointly fail to support an answer.<n>We introduce CARGO-VL, a group-relative framework that optimize matched variants covering aligned, image-correct, text-correct, and both-wrong (A/V/T/N) evidence states.<n>We also contribute XMC (eXtended Modal Conflict), a four-condition conflict training resource, and evaluate transfer on CMC-Bench and Modality-Bias.
  arXiv  Detail & Related papers  (2026-08-05T06:47:46Z)
• Modality Agreement- and Conflict-Aware Prototype Hypergraph Learning for Multimodal Intent Understanding [2.2012643583422347]
  We propose a hierarchical prototype-hypergraph framework that represents multimodal agreement and conflict as distinct, recurring relational structures.<n>MACH (Modality Agreement- and Conflict-aware prototype Hypergraph) represents multimodal agreement and conflict as distinct, recurring relational structures.
  arXiv  Detail & Related papers  (2026-08-04T10:55:51Z)
• Regime-Aware Peer Specialization for Robust RAG under Heterogeneous Knowledge Conflicts [46.1469246256932]
  We propose RAPS-DA, a regime-aware peer specialization framework.<n>At the sample level, conflicts are divided into three regimes, including Grounding, Arbitration, and Resistance.<n>At the token level, a dual-layer selector uses inter-teacher disagreement, student-teacher divergence, and student entropy to filter uninformative or unstable tokens.
  arXiv  Detail & Related papers  (2026-06-29T16:25:13Z)
• ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence [48.938290419832335]
  We introduce ConflictScore, a novel metric that quantifies how well a model's response acknowledges conflicting evidence in its grounding documents.<n>Our framework decomposes responses into atomic claims, labels each claim against each grounding document, and then aggregates these labels into two complementary measures: ConflictScore-Count (CS-C), the proportion of claims exhibiting conflicts, and ConflictScore-Ratio (CS-R), the balance between supporting and contradicting evidence.
  arXiv  Detail & Related papers  (2026-06-24T23:00:09Z)
• Mitigating Context-Memory Conflicts in LLMs through Dynamic Cognitive Reconciliation Decoding [51.242997961841276]
  Large language models accumulate parametric knowledge through pre-training.<n> knowledge conflicts occur when outdated or incorrect parametric knowledge conflicts with external knowledge in the context.<n>Existing methods address knowledge conflicts through contrastive decoding, but in conflict-free scenarios, static approaches disrupt output distribution.<n>We propose Dynamic Cognitive Reconciliation Decoding (DCRD), to predict and context-memory conflicts.
  arXiv  Detail & Related papers  (2026-05-12T14:29:45Z)
• URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection [17.19666367498091]
  Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image.<n>We propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during interaction and fusion.<n>URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines.
  arXiv  Detail & Related papers  (2026-04-08T06:50:43Z)
• Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning [78.86309644343295]
  Multimodal large language models (MLLMs) in long chain-of-thought reasoning often fail when different knowledge sources provide conflicting signals.<n>We formalize these failures under a unified notion of knowledge conflict, distinguishing input-level objective conflict from process-level effective conflict.<n>Our findings provide a mechanism-level view of multimodal reasoning under knowledge conflict and enable principled diagnosis and control of long-CoT failures.
  arXiv  Detail & Related papers  (2026-02-16T07:10:44Z)
• CLASH: A Benchmark for Cross-Modal Contradiction Detection [15.134491772506196]
  CLASH is a novel benchmark for multimodal contradiction detection.<n>It features COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions.
  arXiv  Detail & Related papers  (2025-11-24T15:09:07Z)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning [49.17801010041155]
  Multimodal large language models (MLLMs) promise enhanced reasoning by integrating diverse inputs such as text, vision, and audio.<n>Yet cross-modal reasoning remains underexplored, with conflicting reports on whether added modalities help or harm performance.<n>We categorize multimodal reasoning into six interaction patterns, varying how facts are distributed across modalities and logically combined.
  arXiv  Detail & Related papers  (2025-09-28T08:46:11Z)
• CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation [24.952907733127223]
  We propose a general framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD)<n>CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates mismatchs while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio)
  arXiv  Detail & Related papers  (2025-05-21T08:11:07Z)
• Multi-level Conflict-Aware Network for Multi-modal Sentiment Analysis [6.168616672805216]
  We propose a novel multi-level conflict-aware network (MCAN) for multimodal sentiment analysis.<n>MCAN progressively segregates alignment and conflict constituents from unimodal and bimodal representations.<n>In the conflict modeling branch, we conduct discrepancy constraints at both the representation and predicted output levels.
  arXiv  Detail & Related papers  (2025-02-13T09:14:36Z)
• AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge [57.66282463340297]
  Knowledge conflict arises from discrepancies between information in the context of a large language model and the knowledge stored in its parameters.<n>We propose a fine-grained, instance-level approach called AdaCAD, which dynamically infers the weight of adjustment based on the degree of conflict.<n>We show that ADACAD consistently outperforms other decoding baselines with average QA accuracy gains of 14.21% (absolute) over a static contrastive baseline, and improves the factuality of summaries by 6.19 (AlignScore)
  arXiv  Detail & Related papers  (2024-09-11T16:35:18Z)
• Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition [69.32305810128994]
  Incongruity between modalities poses a challenge for multimodal fusion, especially in affect recognition. We propose the Hierarchical Crossmodal Transformer with Dynamic Modality Gating (HCT-DMG), a lightweight incongruity-aware model. HCT-DMG: 1) outperforms previous multimodal models with a reduced size of approximately 0.8M parameters; 2) recognizes hard samples where incongruity makes affect recognition difficult; 3) mitigates the incongruity at the latent level in crossmodal attention.
  arXiv  Detail & Related papers  (2023-05-23T01:24:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.