Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations

• URL: http://arxiv.org/abs/2608.03611v1
• Date: Tue, 04 Aug 2026 13:02:48 GMT
• Title: Rethinking Modality Reliability in Multimodal Sentiment Analysis with Incomplete Observations
• Abstract summary: We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues.<n>We propose Modality Reliability-Calibrated Framework for Multimodal Sentiment Analysis with incomplete observations.<n> MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency.<n>Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols.
• Score: 9.993152436695421
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Sentiment Analysis (MSA) integrates text, audio, and vision to infer human affect, yet real-world multimodal observations are often incomplete. Existing methods for incomplete-observation MSA mainly follow two paradigms. Reconstruction-based methods recover missing information from observed modalities, while joint-representation methods learn directly from incomplete inputs. Although effective, these methods usually treat modality reliability only implicitly within representation learning or fusion design rather than modeling it explicitly. We argue that modality reliability is a central variable in incomplete-observation settings. Failure to model it explicitly gives rise to two related issues. The first is reliability mismatch, in which the affective evidence retained by each modality varies across samples and missing rates. The second is reliability propagation bias, in which messages from degraded modalities may adversely affect cross-modal interaction and predictive performance. To address these issues, we propose MRCF, a Modality Reliability-Calibrated Framework for MSA with incomplete observations. MRCF contains a Reliability-Aware Branch that estimates sample-specific modality reliability from intramodal quality cues and cross-modal semantic consistency, a Reliability-Guided Interaction Branch that uses the estimated scores to modulate cross-modal information flow, and a Reliability-Calibrated Fusion Module that integrates reliability and semantic cues for final prediction. Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS show that MRCF achieves strong performance under standard incomplete-observation protocols. Further analyses provide evidence that explicit reliability modeling helps mitigate reliability mismatch and reliability propagation bias during interaction and fusion.

Related papers

• Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition [2.2012643583422347]
  Multimodal intent recognition combines linguistic, acoustic, and visual evidence, but individual modalities may be noisy, missing, semantically conflicting, or disproportionately dominant.<n>We propose PRIME, a closed-loop reliability guided framework that jointly diagnoses, restores, and reassesses modality quality at the sample level.<n> Experiments on multimodal intent-recognition benchmarks show that PRIME maintains competitive clean-data performance while improving under missing, noisy, conflicting, and modality-imbalanced conditions.
  arXiv  Detail & Related papers  (2026-08-04T11:14:13Z)
• Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment [53.50824432699408]
  Traits Run Deeper is a novel personality assessment framework.<n>It consists of three components: the Multimodal Foundation Representation (MFR), the Trait-Specific Modality Fusion (TSMF) and the Distribution-Calibrated Personality Regression (DCPR)
  arXiv  Detail & Related papers  (2026-06-09T06:38:36Z)
• Towards Trustworthy Depression Estimation via Disentangled Evidential Learning [50.22167852149165]
  We propose EviDep, an evidential learning framework that jointly quantifies depression severity.<n>EviDep enforces strict information integrity to guarantee robust evidence synthesis.<n>It achieves state-of-the-art predictive accuracy and superior uncertainty calibration, delivering a robust fail-safe mechanism for trustworthy clinical screening.
  arXiv  Detail & Related papers  (2026-04-17T13:27:11Z)
• URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection [17.19666367498091]
  Multimodal sarcasm detection (MSD) aims to identify sarcastic intent from semantic incongruity between text and image.<n>We propose Uncertainty-aware Robust Multimodal Fusion (URMF), a unified framework that explicitly models modality reliability during interaction and fusion.<n>URMF consistently outperforms strong unimodal, multimodal, and MLLM-based baselines.
  arXiv  Detail & Related papers  (2026-04-08T06:50:43Z)
• QA-MoE: Towards a Continuous Reliability Spectrum with Quality-Aware Mixture of Experts for Robust Multimodal Sentiment Analysis [9.551550608808752]
  We introduce a Continuous Reliability Spectrum to unify missingness and quality degradation into a single framework.<n>We then propose QA-MoE, a Quality-Aware Mixture-of-Experts framework that quantifies modality reliability via self-supervised aleatoric uncertainty.<n>Extensive experiments indicate that QA-MoE achieves competitive or state-of-the-art performance across diverse degradation scenarios.
  arXiv  Detail & Related papers  (2026-04-07T11:04:23Z)
• Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities [26.613566093134065]
  Multimodal Sentiment Analysis seeks to infer human emotions by integrating textual, acoustic, and visual cues.<n>Existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities.<n>We propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2026-03-10T02:45:02Z)
• ERA: Evidence-based Reliability Alignment for Honest Retrieval-Augmented Generation [19.616905640490252]
  We propose a new framework called ERA (Evidence-based Reliability Alignment) to enhance abstention behavior in RAG systems.<n>Our approach significantly outperforms baselines, optimizing the trade-off between answer coverage and abstention with superior calibration.
  arXiv  Detail & Related papers  (2026-02-24T01:45:51Z)
• Evidential Deep Partial Multi-View Classification With Discount Fusion [24.139495744683128]
  We propose a novel framework called Evidential Deep Partial Multi-View Classification (EDP-MVC) We use K-means imputation to address missing views, creating a complete set of multi-view data. The potential conflicts and uncertainties within this imputed data can affect the reliability of downstream inferences.
  arXiv  Detail & Related papers  (2024-08-23T14:50:49Z)
• Confidence-aware multi-modality learning for eye disease screening [58.861421804458395]
  We propose a novel multi-modality evidential fusion pipeline for eye disease screening. It provides a measure of confidence for each modality and elegantly integrates the multi-modality information. Experimental results on both public and internal datasets demonstrate that our model excels in robustness.
  arXiv  Detail & Related papers  (2024-05-28T13:27:30Z)
• Reliable Multimodality Eye Disease Screening via Mixture of Student's t Distributions [49.4545260500952]
  We introduce a novel multimodality evidential fusion pipeline for eye disease screening, EyeMoSt. Our model estimates both local uncertainty for unimodality and global uncertainty for the fusion modality to produce reliable classification results. Our experimental findings on both public and in-house datasets show that our model is more reliable than current methods.
  arXiv  Detail & Related papers  (2023-03-17T06:18:16Z)
• Trusted Multi-View Classification with Dynamic Evidential Fusion [73.35990456162745]
  We propose a novel multi-view classification algorithm, termed trusted multi-view classification (TMC) TMC provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. Both theoretical and experimental results validate the effectiveness of the proposed model in accuracy, robustness and trustworthiness.
  arXiv  Detail & Related papers  (2022-04-25T03:48:49Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.