Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?

• URL: http://arxiv.org/abs/2607.17262v1
• Date: Sun, 19 Jul 2026 14:07:43 GMT
• Title: Should Missing Modalities Always Be Necessary to Repair for Multi-modal Sentiment Analysis?
• Abstract summary: Existing methods for multimodal sentiment analysis under missing modalities usually follow a repair-first paradigm.<n>We propose textbfSufficiency-textbfInformed textbfEvidential textbfValtextbfvE (textbfSIEVE) that turns whether to repair'' into an explicit, learnable decision.
• Score: 44.69722591655738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing methods for multimodal sentiment analysis (MSA) under missing modalities usually follow a repair-first paradigm. We revisit this assumption and ask: \emph{should every missing modality be repaired?} A per-sample oracle analysis shows the answer is not always: full-modality input is optimal for only a small fraction of samples, and every modality subset is preferred by some samples. These results suggest that adding or repairing modalities may not always improve prediction, and that the utility of each modality is sample-dependent. Building on this finding, we propose \textbf{S}ufficiency-\textbf{I}nformed \textbf{E}vidential \textbf{V}al\textbf{vE} (\textbf{SIEVE}) that turns ``whether to repair'' into an explicit, learnable decision at the sample level. SIEVE compares a direct prediction branch with a repair branch, derives an empirical sufficiency signal from their per-sample loss gap, and routes each input through an evidential gate that jointly models sufficiency and its epistemic uncertainty. SIEVE is repair-agnostic: it operates as a plug-and-play decision on top of any explicit or implicit repair module, without modifying its internal design. Experiments on CMU-MOSI and IEMOCAP show that SIEVE consistently improves representative repair backbones across evaluated missing rates, and approaches the per-sample dual-branch achievable optimum.

Related papers

• VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following [11.967752833065655]
  Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging.<n>We present VISA, an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop.<n> Experiments on MM-IFEval show that VISA consistently improves multimodal instruction following over strong baselines.
  arXiv  Detail & Related papers  (2026-08-26T16:55:58Z)
• Conformal Fusion Under Missing Modalities [6.599344783327054]
  We argue that missing-modality robustness and calibrated uncertainty are a single coupled property, and introduce Modality-Conditioned Conformal Fusion (MCCF)<n>MCCF combines a multimodal bottleneck fusion backbone trained with modality dropout, per-modality evidential heads producing modality-decomposed Dirichlet distributions, and a Dempster-Shafer combination rule.<n>A Mondrian conformal calibration module keyed on the modality-presence mask then provides finite-sample group-conditional coverage for every non-empty modality subset.
  arXiv  Detail & Related papers  (2026-08-07T12:57:16Z)
• GAUGE: Granularity-Adaptive Counterfactual Gating of Evidence for Incomplete Multimodal Classification [17.595478415811687]
  Multimodal classification typically assumes all modalities are available, yet real-world inputs are often incomplete.<n>We propose GAUGE, a lightweight counterfactual gating framework for incomplete multimodal classification.<n>GAUGE scores the counterfactual effect of replacing every unit with a reference representation through prediction-aware Taylor evidence scores.
  arXiv  Detail & Related papers  (2026-08-06T05:07:14Z)
• Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition [2.2012643583422347]
  Multimodal intent recognition combines linguistic, acoustic, and visual evidence, but individual modalities may be noisy, missing, semantically conflicting, or disproportionately dominant.<n>We propose PRIME, a closed-loop reliability guided framework that jointly diagnoses, restores, and reassesses modality quality at the sample level.<n> Experiments on multimodal intent-recognition benchmarks show that PRIME maintains competitive clean-data performance while improving under missing, noisy, conflicting, and modality-imbalanced conditions.
  arXiv  Detail & Related papers  (2026-08-04T11:14:13Z)
• Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI [3.7524666944787644]
  Two questions matter beyond the size of the accuracy loss: which modality was responsible, and whether the model fails loudly or silently once that modality is dropped.<n>We present a model-agnostic modality-failure framework: given N modality embeddings, any mask-aware probe, and labels, it returns a per-example failure taxonomy.<n>It reports per-modality loud-vs-silent rates, and scales to a three-modality complementarity matrix.
  arXiv  Detail & Related papers  (2026-08-02T19:38:56Z)
• Prototype-Anchored Generalized Manifold Regression for Unknown-Domain Object Detection [58.25418970608328]
  We study Single-Domain Generalized Object Detection (Single-DGOD), which aims to transfer a detector trained on a single source domain to multiple unseen domains.<n>Existing methods mainly rely on simulation-driven strategies, such as data augmentation or textual prompts, to enlarge the training distribution.<n>We propose Manifold Regression with Visual-Text Dual Chain-of-Thought (MR-DCoT), which formulates unknown-domain generalization as a manifold regression problem.
  arXiv  Detail & Related papers  (2026-07-08T09:27:15Z)
• Rethinking Training & Inference for Forecasting: Linking Winner-Take-All back to GMMs [48.245159309501666]
  Trajectory forecasting for autonomous driving has advanced rapidly, yet representative models often produce uninformative posteriors over forecast modes.<n>We argue that this K-means-like hard assignment (one-hot) is the source of uninformative mode probabilities.<n>We introduce two post-hoc treatments: (1) test-time posterior-weighted merging that aggregates nearby candidate trajectories; and (2) a one-step expectation-maximization (EM) update.
  arXiv  Detail & Related papers  (2026-06-24T22:26:43Z)
• Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization [14.039980020878815]
  Modern language-model fine-tuning pairs each prompt with a single response, even though many prompts admit multiple valid completions.<n>This effectively reduces a multi-modal conditional distribution to a one-sample view, where training emphasizes a subset of plausible modes.<n>We study multi-response training (MRT), which retains multiple responses per prompt, and develop a principled account of when and why it helps.
  arXiv  Detail & Related papers  (2026-05-30T05:30:39Z)
• Knowing When Not to Answer: Evaluating Abstention in Multimodal Reasoning Systems [52.83669998269706]
  Abstention has been studied in text-only settings but remains underexplored multimodally.<n>Current benchmarks either ignore unanswerability or rely on coarse methods that miss realistic failure modes.<n>We introduce MM-AQA, a benchmark that constructs unanswerable instances from answerable ones via transformations along two axes.
  arXiv  Detail & Related papers  (2026-04-16T09:23:22Z)
• CUPID: A Plug-in Framework for Joint Aleatoric and Epistemic Uncertainty Estimation with a Single Model [1.6855257501750571]
  CUPID (Comprehensive Uncertainty Plug-in estImation moDel) is a general-purpose module that jointly estimates aleatoric and epistemic uncertainty without modifying or retraining the base model.<n>We evaluate CUPID across a range of tasks, including classification, regression, and out-of-distribution detection.<n>The results show that CUPID consistently delivers competitive performance while offering layer-wise insights into the origins of uncertainty.
  arXiv  Detail & Related papers  (2026-03-11T13:18:10Z)
• Nearly Optimal Bayesian Inference for Structural Missingness [23.988531482641307]
  In the Bayesian view, prediction via the posterior predictive distribution integrates over the full model posterior uncertainty.<n>This framework decouples learning an in-model missing-value posterior from (ii) label prediction by optimizing the predictive posterior distribution.<n>It achieves SOTA on 43 classification and 15 imputation benchmarks, with finite-sample near Bayes-optimality guarantees.
  arXiv  Detail & Related papers  (2026-01-26T14:03:11Z)
• Calibrated Multimodal Representation Learning with Missing Modalities [100.55774771852468]
  Multimodal representation learning harmonizes distinct modalities by aligning them into a unified latent space.<n>Recent research generalizes traditional cross-modal alignment to produce enhanced multimodal synergy but requires all modalities to be present for a common instance.<n>We provide theoretical insights into this issue from an anchor shift perspective.<n>We propose CalMRL for multimodal representation learning to calibrate incomplete alignments caused by missing modalities.
  arXiv  Detail & Related papers  (2025-11-15T05:01:43Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.