Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities

• URL: http://arxiv.org/abs/2605.16889v1
• Date: Sat, 16 May 2026 09:03:31 GMT
• Title: Controlling Decision Drift in Multimodal Sentiment Analysis with Missing Modalities
• Abstract summary: We propose a two-level reference alignment framework to improve robustness under modality missing.<n>First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space.<n>Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting.
• Score: 20.8430910234166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal sentiment analysis relies on textual, acoustic, and visual signals, yet real-world data often suffer from modality missing and quality imbalance. Existing methods generate features for modality missing from available ones, but differences in expression mechanisms and sentiment dynamics across modalities may cause the generated features to deviate from true distributions and mislead prediction. In addition, unreliable modalities may dominate fusion, resulting in representation shift across modality combinations and unstable sentiment representations. To address these challenges, we propose a two-level reference alignment framework. The framework introduces stable references at the feature representation and sentiment decision levels to improve robustness under modality missing. First-level reference alignment leverages complete-modality samples to constrain representations and align different modality combinations into a shared sentiment space. Second-level reference alignment enforces cross-modal consistency at the decision level by suppressing unreliable modalities through prototype retrieval and voting. As a result, the framework maintains stable and reliable sentiment predictions under diverse missing-modality patterns. Experiments on CMU-MOSI and CMU-MOSEI show consistent improvements across various missing-modality settings. Under full-modality input, the proposed method achieves state-of-the-art performance, with ACC of 86.28% and 85.88%, and F1 of 86.24% and 85.86%.

Related papers

• Dual-Edged Homogeneous-Modality Similarity: Towards Visible-Infrared Modality-Incomplete Person Re-Identification with Modality Adaptive Matching [60.86381869046298]
  We formalize the Visible-Infrared Modality-Incomplete Re-Identification (VIMI-ReID) task.<n>The unpredictable modality combinations and inherent similarity of homogeneous-modality samples in VIMI-ReID cause a significant performance drop.<n>We propose the Modality Adaptive Matching Transformer (MAMT) to extract modality-specific and modality-shared features.
  arXiv  Detail & Related papers  (2026-07-21T04:10:20Z)
• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach [52.92973028971813]
  We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition.<n>The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model.<n>Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality.
  arXiv  Detail & Related papers  (2026-07-16T08:04:23Z)
• Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment [41.72676356491381]
  Action Quality Assessment (AQA) aims to evaluate how well a person performs a movement.<n>Unimodal approaches often struggle to capture subtle cues of movement quality in real-world settings.<n>We propose DualAlign, a two-stage multi-modal fusion framework with adaptive alignment.
  arXiv  Detail & Related papers  (2026-07-08T14:09:13Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection [17.19666367498091]
  Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image.<n>We propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during interaction and fusion.<n>URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines.
  arXiv  Detail & Related papers  (2026-04-08T06:50:43Z)
• Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities [21.767502810187477]
  The missing modality problem poses a fundamental challenge in multimodal sentiment analysis.<n>Existing approaches primarily improve robustness through prompt learning and pre trained models.<n>A Prompt based Missing Modality Adaptation framework is proposed to address these issues.
  arXiv  Detail & Related papers  (2026-04-07T07:59:06Z)
• Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities [26.613566093134065]
  Multimodal Sentiment Analysis seeks to infer human emotions by integrating textual, acoustic, and visual cues.<n>Existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities.<n>We propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2026-03-10T02:45:02Z)
• SGMA: Semantic-Guided Modality-Aware Segmentation for Remote Sensing with Incomplete Multimodal Data [31.146366498415784]
  Multimodal semantic segmentation integrates complementary information from diverse sensors for remote sensing Earth observation.<n>IMSS faces three key challenges: multimodal imbalance, where dominant modalities suppress fragile ones; intra-class variation in scale, shape, and orientation across modalities; and cross-modal heterogeneity with conflicting cues producing inconsistent semantic responses.<n>We propose the Semantic-Guided Modality-Aware (SGMA) framework, which ensures balanced multimodal learning while reducing intra-class variation and reconciling cross-modal inconsistencies through semantic guidance.
  arXiv  Detail & Related papers  (2026-03-03T01:28:21Z)
• Benchmarking Multi-modal Semantic Segmentation under Sensor Failures: Missing and Noisy Modality Robustness [61.87055159919641]
  Multi-modal semantic segmentation (MMSS) addresses the limitations of single-modality data by integrating complementary information across modalities.<n>Despite notable progress, a significant gap persists between research and real-world deployment due to variability and uncertainty in multi-modal data quality.<n>We introduce a robustness benchmark that evaluates MMSS models under three scenarios: Entire-Missing Modality (EMM), Random-Missing Modality (RMM), and Noisy Modality (NM)
  arXiv  Detail & Related papers  (2025-03-24T08:46:52Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.