Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

• URL: http://arxiv.org/abs/2606.30597v1
• Date: Mon, 29 Jun 2026 17:36:47 GMT
• Title: Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities
• Abstract summary: Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities.<n>In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation.<n>Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features.<n>As higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities. In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation. Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features, regardless of whether the input modalities are complete or partially absent. However, such input-conditioned strategies are hindered by the escalating unreliability of instance-level features; as higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance. To address this limitation, we hypothesize that learnable latent prompts themselves encapsulate stable, modality-intrinsic priors that are decoupled from corrupted inputs. Consequently, we propose a novel paradigm: Learning from Reliable Latent Prompts. Unlike prior methods, we model input-agnostic learnable prompts as stable latent anchors that enable robust guidance and effective cross-modal knowledge compensation, even under extreme missing rates (e.g., 90%). Empirical results across three benchmark datasets demonstrate that our "learn-from-latent-prompts" approach achieves state-of-the-art performance across a wide range of missing-modality scenarios. Extensive experiments further confirm the effectiveness of this paradigm in providing a robust solution to the missing-modality problem.

Related papers

• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities [21.767502810187477]
  The missing modality problem poses a fundamental challenge in multimodal sentiment analysis.<n>Existing approaches primarily improve robustness through prompt learning and pre trained models.<n>A Prompt based Missing Modality Adaptation framework is proposed to address these issues.
  arXiv  Detail & Related papers  (2026-04-07T07:59:06Z)
• Synergistic Prompting for Robust Visual Recognition with Missing Modalities [13.821274074204082]
  Large-scale multi-modal models have demonstrated remarkable performance across various visual recognition tasks.<n>The presence of missing or incomplete modality inputs often leads to significant performance degradation.<n>We propose a novel Synergistic Prompting framework for robust visual recognition with missing modalities.
  arXiv  Detail & Related papers  (2025-07-10T14:28:12Z)
• Decoupled Multimodal Prototypes for Visual Recognition with Missing Modalities [3.88369051454137]
  Multimodal learning enhances deep learning models by enabling them to perceive and understand information from multiple data modalities.<n>Most existing approaches assume the availability of all modalities, an assumption that often fails in real-world applications.<n>Recent works have introduced learnable missing-case-aware prompts to mitigate performance degradation caused by missing modalities.<n>We propose a novel decoupled prototype-based output head, which leverages missing-case-aware class-wise prototypes tailored for each individual modality.
  arXiv  Detail & Related papers  (2025-05-13T06:53:37Z)
• Deep Correlated Prompting for Visual Recognition with Missing Modalities [22.40271366031256]
  Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty. We refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input.
  arXiv  Detail & Related papers  (2024-10-09T05:28:43Z)
• Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models [6.610033827647869]
  In real-world scenarios, consistently acquiring complete multimodal data presents significant challenges. This often leads to the issue of missing modalities, where data for certain modalities are absent. We propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method.
  arXiv  Detail & Related papers  (2024-07-17T14:44:25Z)
• Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition [52.522244807811894]
  We propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters.
  arXiv  Detail & Related papers  (2024-07-07T13:55:56Z)
• Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos [92.38662956154256]
  Real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues.<n>We propose a novel approach to address this issue at test time without requiring retraining.<n>MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time.
  arXiv  Detail & Related papers  (2024-04-23T16:01:33Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• Self-attention fusion for audiovisual emotion recognition with incomplete data [103.70855797025689]
  We consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms.
  arXiv  Detail & Related papers  (2022-01-26T18:04:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.