Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models

• URL: http://arxiv.org/abs/2608.17306v1
• Date: Tue, 18 Aug 2026 03:00:27 GMT
• Title: Learning What Not to Learn: Adversarial Disentangled Prompt Tuning for Robust Vision-Language Models
• Abstract summary: We propose ADAPT (Adversarial Disentangled Prompt Tuning) as a robust prompt tuning framework.<n>By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting.<n> Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes.
• Score: 15.218496545665692
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts). To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of ``Learning What Not to Learn''. Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features. By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes. The code is available at https://github.com/cheny02/ADAPT-ACMMM2026.

Related papers

• Training Prompt Matters: State-Adaptive Optimization for Robust Fine-Tuning [14.317667911880482]
  We show that paraphrased prompts induce drastically different cross-task impacts regarding forgetting and generalization.<n>These impacts are positively correlated across tasks, indicating the existence of superior prompts that consistently yield better performance.<n>We introduce State-Adaptive Prompt Optimization (SAPO), a lightweight yet effective training strategy that shifts task formulation from a static input to a dynamic, state-adaptive variable.
  arXiv  Detail & Related papers  (2026-06-01T09:31:28Z)
• The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning [54.67958855362658]
  We propose a dynamic framework that stress tests unlearning using complex structured queries.<n>Our approach first elicits knowledge from the target model (pre-unlearning) and constructs targeted probes, ranging from simple queries to multi-hop chains.<n>Our framework enables practical and scalable evaluation of unlearning methods without the need for manual construction of forget test sets.
  arXiv  Detail & Related papers  (2026-03-11T19:51:33Z)
• Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment [6.168302049077305]
  Inoculation Prompting prevents learning of an undesired behavior by modifying training prompts to explicitly request it.<n>We find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities.
  arXiv  Detail & Related papers  (2025-10-06T17:02:59Z)
• Robust Anti-Backdoor Instruction Tuning in LVLMs [53.766434746801366]
  We introduce a lightweight, certified-agnostic defense framework for large visual language models (LVLMs)<n>Our framework finetunes only adapter modules and text embedding layers under instruction tuning.<n>Experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero.
  arXiv  Detail & Related papers  (2025-06-04T01:23:35Z)
• Mixture of Experts Meets Prompt-Based Continual Learning [23.376460019465235]
  This paper conducts a theoretical analysis to unravel how prompts bestow such advantages in continual learning.<n>We provide a novel view on prefix tuning, reframing it as the addition of new task-specific experts, thereby inspiring the design of a novel gating mechanism.<n>The effectiveness of NoRGa is substantiated both theoretically and empirically across diverse benchmarks and pretraining paradigms.
  arXiv  Detail & Related papers  (2024-05-23T02:49:57Z)
• RESTORE: Towards Feature Shift for Vision-Language Prompt Learning [33.13407089704543]
  We show that prompt tuning along only one branch of CLIP is the reason why the misalignment occurs. Without proper regularization across the learnable parameters in different modalities, prompt learning violates the original pre-training constraints. We propose RESTORE, a multi-modal prompt learning method that exerts explicit constraints on cross-modal consistency.
  arXiv  Detail & Related papers  (2024-03-10T08:52:48Z)
• Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model [86.9619638550683]
  Vision-language foundation models have exhibited remarkable success across a multitude of downstream tasks due to their scalability on extensive image-text paired data.<n>However, these models display significant limitations when applied to downstream tasks, such as fine-grained image classification, as a result of decision shortcuts''
  arXiv  Detail & Related papers  (2024-03-01T09:01:53Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting [112.66832145320434]
  We introduce a self-regularization framework for prompting called PromptSRC. PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
  arXiv  Detail & Related papers  (2023-07-13T17:59:35Z)
• Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models [108.13378788663196]
  We propose Subspace Prompt Tuning (SubPT) to project the gradients in back-propagation onto the low-rank subspace spanned by the early-stage gradient flow eigenvectors during the entire training process. We equip CoOp with Novel Learner Feature (NFL) to enhance the generalization ability of the learned prompts onto novel categories beyond the training set.
  arXiv  Detail & Related papers  (2022-11-04T02:06:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.