Dropout Prompt Learning: Towards Robust and Adaptive Vision-Language Models

• URL: http://arxiv.org/abs/2512.07234v1
• Date: Mon, 08 Dec 2025 07:31:27 GMT
• Title: Dropout Prompt Learning: Towards Robust and Adaptive Vision-Language Models
• Authors: Biao Chen, Lin Zuo, Mengmeng Jing, Kunbin He, Yuchen Wang,
• Abstract summary: Dropout Prompt Learning aims for applying dropout to improve the robustness of vision-language models.<n>Our method surpasses regularization-based methods including KgCoOp by 5.10% and PromptSRC by 2.13% in performance on base-to-novel generalization.
• Score: 22.573044825857043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dropout is a widely used regularization technique which improves the generalization ability of a model by randomly dropping neurons. In light of this, we propose Dropout Prompt Learning, which aims for applying dropout to improve the robustness of the vision-language models. Different from the vanilla dropout, we apply dropout on the tokens of the textual and visual branches, where we evaluate the token significance considering both intra-modal context and inter-modal alignment, enabling flexible dropout probabilities for each token. Moreover, to maintain semantic alignment for general knowledge transfer while encouraging the diverse representations that dropout introduces, we further propose residual entropy regularization. Experiments on 15 benchmarks show our method's effectiveness in challenging scenarios like low-shot learning, long-tail classification, and out-of-distribution generalization. Notably, our method surpasses regularization-based methods including KgCoOp by 5.10% and PromptSRC by 2.13% in performance on base-to-novel generalization.

Related papers

• Semantic-guided Fine-tuning of Foundation Model for Long-tailed Visual Recognition [38.74388860692423]
  We propose a novel approach, Semantic-guided fine-tuning of foundation model for long-tailed visual recognition (Sage)<n>We introduce an SG-Adapter that integrates class descriptions as semantic guidance to guide the fine-tuning of the visual encoder.<n>Experiments on benchmark datasets demonstrate the effectiveness of the proposed Sage in enhancing performance in long-tailed learning.
  arXiv  Detail & Related papers  (2025-07-17T05:47:19Z)
• AttentionDrop: A Novel Regularization Method for Transformer Models [0.3262230127283452]
  Transformer-based architectures achieve state-of-the-art performance across a wide range of tasks in natural language processing, computer vision, and speech processing.<n>However, their immense capacity often leads to overfitting, especially when training data is limited or noisy.<n>This research proposes a unified family of regularization techniques, which operate directly on the self-attention distributions.
  arXiv  Detail & Related papers  (2025-04-16T13:51:16Z)
• Efficient Transfer Learning for Video-language Foundation Models [13.166348605993292]
  We propose a parameter-efficient Multi-modalpatio Ssupervised-Temporal Adapter (MSTA) to enhance alignment between textual and visual representations.<n>We evaluate the effectiveness of our approach across four tasks: zero-shot transfer, few-shot learning, base-to-novel generalization, and fully-Temporal learning.
  arXiv  Detail & Related papers  (2024-11-18T01:25:58Z)
• Diffusing States and Matching Scores: A New Framework for Imitation Learning [16.941612670582522]
  Adversarial Imitation Learning is traditionally framed as a two-player zero-sum game between a learner and an adversarially chosen cost function.<n> diffusion models have emerged as a non-adversarial alternative to GANs that merely require training a score function via regression.<n>We show our approach outperforms both GAN-style imitation learning baselines and discriminator-free imitation learning baselines across various continuous control problems.
  arXiv  Detail & Related papers  (2024-10-17T17:59:25Z)
• Collapsed Language Models Promote Fairness [88.48232731113306]
  We find that debiased language models exhibit collapsed alignment between token representations and word embeddings.<n>We design a principled fine-tuning method that can effectively improve fairness in a wide range of debiasing methods.
  arXiv  Detail & Related papers  (2024-10-06T13:09:48Z)
• Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification [49.41632476658246]
  We discuss the extension of DFKD to Vision-Language Foundation Models without access to the billion-level image-text datasets. The objective is to customize a student model for distribution-agnostic downstream tasks with given category concepts. We propose three novel Prompt Diversification methods to encourage image synthesis with diverse styles.
  arXiv  Detail & Related papers  (2024-07-21T13:26:30Z)
• Few-Shot Adversarial Prompt Learning on Vision-Language Models [62.50622628004134]
  The vulnerability of deep neural networks to imperceptible adversarial perturbations has attracted widespread attention. Previous efforts achieved zero-shot adversarial robustness by aligning adversarial visual features with text supervision. We propose a few-shot adversarial prompt framework where adapting input sequences with limited data makes significant adversarial robustness improvement.
  arXiv  Detail & Related papers  (2024-03-21T18:28:43Z)
• POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained models [62.23255433487586]
  We propose an unsupervised fine-tuning framework to fine-tune the model or prompt on the unlabeled target data. We demonstrate how to apply our method to both language-augmented vision and masked-language models by aligning the discrete distributions extracted from the prompts and target data.
  arXiv  Detail & Related papers  (2023-04-29T22:05:22Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Explaining, Evaluating and Enhancing Neural Networks' Learned Representations [2.1485350418225244]
  We show how explainability can be an aid, rather than an obstacle, towards better and more efficient representations. We employ such attributions to define two novel scores for evaluating the informativeness and the disentanglement of latent embeddings. We show that adopting our proposed scores as constraints during the training of a representation learning task improves the downstream performance of the model.
  arXiv  Detail & Related papers  (2022-02-18T19:00:01Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.