T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models

• URL: http://arxiv.org/abs/2606.23132v1
• Date: Mon, 22 Jun 2026 10:21:24 GMT
• Title: T-VSS: Test-Time Visual Subspace Steering for Adversarial Robustness of Vision-Language Models
• Abstract summary: Vision-language models (VLMs) achieve strong zero-shot recognition, but they remain highly vulnerable to adversarial perturbations.<n>Recent test-time adaptations improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself.<n>We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space.
• Score: 30.000286470806355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) achieve strong zero-shot recognition, but they remain highly vulnerable to adversarial perturbations. Recent test-time adaptations improve robustness without retraining, but they do not directly adapt the corrupted visual representation itself. Prompt-based methods adapt the learnable text prompts, while input-space methods optimize pixels or padding at test time. These approaches can improve predictions, but they do so through an indirect and expensive optimization path. We propose Test-time Visual Subspace Steering (T-VSS), a lightweight defense that performs test-time adaptation directly in the visual feature space. T-VSS first builds a sample-specific low-rank subspace from multi-view feature residuals anchored at the attacked image. It then learns a shared feature correction within this subspace using reliability-weighted entropy minimization. By constraining adaptation to a compact visual geometry, T-VSS steers attacked features toward more stable and discriminative predictions while avoiding noisy full-space updates. Experiments on fine-grained, ImageNet, and ImageNet-OOD benchmarks show that T-VSS improves adversarial robustness while maintaining competitive clean accuracy and better efficiency than prior test-time adaptations.

Related papers

• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• Reward-Guided Semantic Evolution for Test-time Adaptive Object Detection [82.2968697030677]
  Open-vocabulary object detection with vision-language models (VLMs) such as Grounding DINO suffers from performance degradation under test-time distribution shifts.<n>We propose Reward-Guided Semantic Evolution (RGSE), a training-free framework that directly refines the text embeddings at test time.
  arXiv  Detail & Related papers  (2026-05-06T06:17:41Z)
• Bootstrapping Video Semantic Segmentation Model via Distillation-assisted Test-Time Adaptation [58.40817037271021]
  DiTTA (Distillation-assisted Test-Time Adaptation) is a novel framework that converts an ISS model into a temporally-aware VSS model without annotated videos.<n>We demonstrate DiTTA's effectiveness, achieving competitive or superior performance relative to fully-supervised VSS methods.
  arXiv  Detail & Related papers  (2026-04-13T03:47:08Z)
• Geometry-Aware Semantic Reasoning for Training Free Video Anomaly Detection [9.337342114768942]
  Training-free video anomaly detection (VAD) has recently emerged as a scalable alternative to supervised approaches.<n>We introduce MM-VAD, a geometry-aware semantic reasoning framework for training free VAD.<n> MM-VAD reframes anomaly detection as adaptive test-time inference rather than fixed feature comparison.
  arXiv  Detail & Related papers  (2026-03-10T09:28:37Z)
• Steering Vision-Language Pre-trained Models for Incremental Face Presentation Attack Detection [62.89126207012712]
  Face Presentation Attack Detection (PAD) demands incremental learning to combat spoofing tactics and domains.<n>Privacy regulations forbid retaining past data, necessitating rehearsal-free learning (RF-IL)
  arXiv  Detail & Related papers  (2025-12-22T04:30:11Z)
• TTP: Test-Time Padding for Adversarial Detection and Robust Adaptation on Vision-Language Models [32.85951917559796]
  We propose Test-Time Padding (TTP), a lightweight defense framework that performs adversarial detection followed by targeted adaptation at inference.<n>TTP consistently surpasses state-of-the-art test-time defenses, delivering substantial improvements in adversarial robustness without compromising clean accuracy.
  arXiv  Detail & Related papers  (2025-12-18T13:34:14Z)
• Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models [34.181702444695794]
  We introduce Spectrum-Aware Test-Time Steering (STS) to steer latent representations in a spectrum-aware manner.<n> STS operates entirely at inference in the latent space, without backpropagation through or modification of the frozen encoders.<n>Building on standard evaluation protocols, our experiments demonstrate that STS largely surpasses or compares favorably against state-of-the-art test-time adaptation methods.
  arXiv  Detail & Related papers  (2025-11-12T23:25:58Z)
• Test-time Loss Landscape Adaptation for Zero-Shot Generalization in Vision-Language Models [3.1099372412393524]
  This paper unveils the unnecessary nature of backpropagation in existing methods from a loss landscape perspective.<n>It proposes a simple yet effective framework called Test-time Loss Landscape Adaptation (TLLA)<n>In the prompt tuning stage, a Sharpness-Aware Prompt Tuning (SAPT) method is introduced to identify the training flat minimum.<n>In the test stage, a Sharpness-based Test Sample Selection (STSS) approach is utilized to ensure the alignment of flat minima.
  arXiv  Detail & Related papers  (2025-01-31T03:10:48Z)
• TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models [53.91006249339802]
  We propose a novel defense method called Test-Time Adversarial Prompt Tuning (TAPT) to enhance the inference robustness of CLIP against visual adversarial attacks. TAPT is a test-time defense method that learns defensive bimodal (textual and visual) prompts to robustify the inference process of CLIP. We evaluate the effectiveness of TAPT on 11 benchmark datasets, including ImageNet and 10 other zero-shot datasets.
  arXiv  Detail & Related papers  (2024-11-20T08:58:59Z)
• Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models [19.683461002518147]
  Test-Time Prototype Shifting (TPS) is a pioneering approach designed to adapt vision-language models to test datasets using unlabeled test inputs.<n>TPS not only facilitates optimization-free prototype reuse for subsequent predictions but also enables seamless integration with current advancements in prompt engineering.<n>A notable aspect of our framework is its significantly reduced memory and computational demands when compared to conventional text-prompt tuning methods.
  arXiv  Detail & Related papers  (2024-03-19T17:54:34Z)
• Visual Prompting for Adversarial Robustness [63.89295305670113]
  We use visual prompting computation to improve adversarial robustness of a fixed, pre-trained model at testing time. We propose a new VP method, termed Class-wise Adrial Visual Prompting (C-AVP), to generate class-wise visual prompts. C-AVP outperforms the conventional VP method, with 2.1X standard accuracy gain and 2X robust accuracy gain.
  arXiv  Detail & Related papers  (2022-10-12T15:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.