SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models

• URL: http://arxiv.org/abs/2606.06943v1
• Date: Fri, 05 Jun 2026 06:12:48 GMT
• Title: SS-TPT: Stability and Suitability-Guided Test-Time Prompt Tuning for Adversarially Robust Vision-Language Models
• Abstract summary: Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations.<n>Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown.<n>We present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT) to evaluate the quality of each augmented view.
• Score: 7.578599270855317
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) such as CLIP achieve strong zero-shot recognition but remain highly fragile under adversarial perturbations. Recent test-time adaptation defenses improve robustness by leveraging many augmented views, but this leads to impractical slowdown and a clear robustness-throughput trade-off. To address this challenge, we present Stability and Suitability-guided Test-time Prompt Tuning (SS-TPT), evaluating the quality of each augmented view via two complementary scores: (1) stability, measuring prediction invariance to weak augmentations, and (2) suitability, measuring feature-space density among views. These stability and suitability (SS) scores guide both adaptation and inference through an SS-guided consistency loss and an SS-weighted prediction, amplifying trustworthy views while suppressing corrupted ones. Extensive experiments demonstrate that SS-TPT significantly outperforms prior state-of-the-art methods, achieving superior robustness-throughput trade-offs across diverse datasets and varying numbers of views, thereby demonstrating both strong practicality and generality. Our code is available at https://github.com/sunoh-kim/SS-TPT.

Related papers

• Beyond Confidence: Stability-Aware Test-Time Adaptation for LLM Reasoning [18.026220495767326]
  Test-time adaptation has emerged as a lightweight alternative to costly post-training for improving Large Language Models (LLMs) on downstream tasks.<n>We propose Test-Time Adaptation via Stability-Aware Confidence Optimization (TASCO), a framework that incorporates local stability into confidence-based test-time adaptation.<n> Experiments demonstrate that TASCO improves reasoning accuracy and token efficiency across diverse LLMs and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-09-10T11:29:18Z)
• Robustifying Vision-Language Models via Test-Time Prompt Adaptation [23.05168102474528]
  We propose RITA, a test-tIme prompt-TAdaptation framework that shifts from sample-level estimates to distribution-level alignment.<n>Specifically, RITA employs optimal transport to align the distribution of augmented visual features with textual prototypes, mitigating adversarial outliers and rectifying cross-modal semantic misalignment.
  arXiv  Detail & Related papers  (2026-07-10T14:19:42Z)
• Reliability-Guided Adaptive Ensembling for Robust Test-Time Adaptation [21.989744597091192]
  Test-time adaptation (TTA) can domain shift without source data, but it is highly brittle under adversarially contaminated test streams.<n>We study robust test-time adaptation (RTTA) in the adversarial-stream setting, which remains comparatively underexplored relative to standard TTA.<n>We propose SAFER, a training-free reliability-guided augmentation wrapper for RTTA.
  arXiv  Detail & Related papers  (2026-06-21T06:20:22Z)
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective [52.693471818837395]
  We argue that PEFT should be assessed through the stability-plasticity dilemma.<n>We introduce PEFT-Arena, a benchmark that measures downstream performance and general capability retention.<n>Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
  arXiv  Detail & Related papers  (2026-05-27T17:59:51Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Less Approximates More: Harmonizing Performance and Confidence Faithfulness via Hybrid Post-Training for High-Stakes Tasks [52.81286869496811]
  Large language models are increasingly deployed in high-stakes tasks, where confident yet incorrect inferences may cause real-world harm.<n>We propose to jointly optimize unsupervised Reinforcement Learning from Internal Feedback with reasoning-trace-guided Reasoning Distillation.<n>We introduce HyTuning, a hybrid post-training framework that adaptively reweights RD and RLIF via a PRG-style metric.
  arXiv  Detail & Related papers  (2026-04-09T16:50:11Z)
• Behavioral Fingerprints for LLM Endpoint Stability and Identity [6.5886705394788665]
  Stability Monitor periodically fingerprints an endpoint by sampling outputs from a fixed prompt set.<n>In controlled validation, Stability Monitor detects changes to model family, version, inference stack, quantization, and behavioral parameters.<n>In real-world monitoring of the same model hosted by multiple providers, we observe substantial provider-to-provider and within-provider stability differences.
  arXiv  Detail & Related papers  (2026-03-19T15:23:23Z)
• Efficient and Stable Reinforcement Learning for Diffusion Language Models [59.75789436018925]
  Reinforcement Learning (RL) is crucial for unlocking the complex reasoning capabilities of Diffusion-based Large Language Models (dLLMs)<n>Applying to dLLMs faces unique challenges in efficiency and stability.<n>We propose Spatio-Temporal Pruning (STP), a framework designed to simultaneously improve the efficiency and stability of RL for dLLMs.
  arXiv  Detail & Related papers  (2026-02-09T17:04:23Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Test-Time Adaptation for Tactile-Vision-Language Models [6.918033886034668]
  Tactile-vision-language (TVL) models are increasingly deployed in real-world robotic and multimodal perception tasks.<n>Existing test-time adaptation methods provide filtering in unimodal settings but lack explicit treatment of modality-wise reliability under cross-modal shifts.<n>We propose a reliability-aware framework that estimates per-modality reliability from prediction uncertainty and perturbation-based responses.
  arXiv  Detail & Related papers  (2026-01-31T02:26:01Z)
• TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models [32.85951917559796]
  We propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference.<n>TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy.
  arXiv  Detail & Related papers  (2025-12-18T13:34:14Z)
• Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations [67.35596444651037]
  Vision-language models (VLMs) exhibit remarkable zero-shot capabilities but struggle with distribution shifts in downstream tasks when labeled data is unavailable.<n>We propose a Reliable Test-time Adaptation (ReTA) method that enhances reliability from two perspectives.
  arXiv  Detail & Related papers  (2025-07-13T05:37:33Z)
• Towards Exemplar-Free Continual Learning in Vision Transformers: an Account of Attention, Functional and Weight Regularization [27.131241302082092]
  We investigate the continual learning of Vision Transformers (ViT) for the challenging exemplar-free scenario. Our work takes an initial step towards a surgical investigation of self-attention mechanism (SAM) We propose an asymmetric variant and apply it to the pooled output distillation (POD) loss adapted for ViTs.
  arXiv  Detail & Related papers  (2022-03-24T16:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.