A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models

• URL: http://arxiv.org/abs/2603.26984v2
• Date: Tue, 31 Mar 2026 12:59:02 GMT
• Title: A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models
• Abstract summary: Energy-Guided Test-Time Transformation (ET3) is a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input samples.<n>Our method is grounded in a theory that proves our transformation succeeds in classification under reasonable assumptions.<n>We present extensive experiments demonstrating that ET3 provides a strong defense for classifiers, zero-shot classification with CLIP, and also for boosting the robustness of LVLMs in tasks such as Image Captioning and Visual Questioning.
• Score: 6.002277724755204
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Despite the rapid progress in multimodal models and Large Visual-Language Models (LVLM), they remain highly susceptible to adversarial perturbations, raising serious concerns about their reliability in real-world use. While adversarial training has become the leading paradigm for building models that are robust to adversarial attacks, Test-Time Transformations (TTT) have emerged as a promising strategy to boost robustness at inference. In light of this, we propose Energy-Guided Test-Time Transformation (ET3), a lightweight, training-free defense that enhances the robustness by minimizing the energy of the input samples. Our method is grounded in a theory that proves our transformation succeeds in classification under reasonable assumptions. We present extensive experiments demonstrating that ET3 provides a strong defense for classifiers, zero-shot classification with CLIP, and also for boosting the robustness of LVLMs in tasks such as Image Captioning and Visual Question Answering. Code is available at github.com/OmnAI-Lab/Energy-Guided-Test-Time-Defense .

Related papers

• ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignment [20.498600810211293]
  We propose a training-free inference method to sample directly from the optimalReinforcement Learning policy.<n>Our algorithm, Energy-Guided Test-Time Scaling (ETS), estimates the key energy term via online Monte Carlo, with a provable convergence rate.<n>ETS substantially reduces inference latency while provably preserving sampling quality.
  arXiv  Detail & Related papers  (2026-01-29T10:06:52Z)
• Adversarial Defense in Vision-Language Models: An Overview [7.668103158377842]
  The widespread use of Vision Language Models (VLMs) has raised concerns about their vulnerability to sophisticated adversarial attacks.<n>To address this challenge, three main defense paradigms have been proposed: Training-time Defense, Test-time Adaptation Defense, and Training-free Defense.
  arXiv  Detail & Related papers  (2026-01-18T14:57:51Z)
• MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models [56.09354775405601]
  Model extraction attacks aim to replicate the functionality of a black-box model through query access.<n>Most existing defenses presume that attacker queries have out-of-distribution (OOD) samples, enabling them to detect and disrupt suspicious inputs.<n>We propose MISLEADER, a novel defense strategy that does not rely on OOD assumptions.
  arXiv  Detail & Related papers  (2025-06-03T01:37:09Z)
• R-TPT: Improving Adversarial Robustness of Vision-Language Models through Test-Time Prompt Tuning [69.72249695674665]
  We propose a robust test-time prompt tuning (R-TPT) for vision-language models (VLMs)<n>R-TPT mitigates the impact of adversarial attacks during the inference stage.<n>We introduce a plug-and-play reliability-based weighted ensembling strategy to strengthen the defense.
  arXiv  Detail & Related papers  (2025-04-15T13:49:31Z)
• Shedding More Light on Robust Classifiers under the lens of Energy-based Models [3.953603590878949]
  We offer a new take on the dynamics of adversarial training (AT) Our analysis of the energy landscape during AT reveals that untargeted attacks generate adversarial images much more in-distribution (lower energy) than the original data from the point of view of the model. Motivated by rigorous evidence, we propose Weighted Energy Adversarial Training (WEAT)
  arXiv  Detail & Related papers  (2024-07-08T18:31:19Z)
• Towards Robust Federated Learning via Logits Calibration on Non-IID Data [49.286558007937856]
  Federated learning (FL) is a privacy-preserving distributed management framework based on collaborative model training of distributed devices in edge networks. Recent studies have shown that FL is vulnerable to adversarial examples, leading to a significant drop in its performance. In this work, we adopt the adversarial training (AT) framework to improve the robustness of FL models against adversarial example (AE) attacks.
  arXiv  Detail & Related papers  (2024-03-05T09:18:29Z)
• Robust Feature Inference: A Test-time Defense Strategy using Spectral Projections [12.807619042576018]
  We propose a novel test-time defense strategy called Robust Feature Inference (RFI) RFI is easy to integrate with any existing (robust) training procedure without additional test-time computation. We show that RFI improves robustness across adaptive and transfer attacks consistently.
  arXiv  Detail & Related papers  (2023-07-21T16:18:58Z)
• Enhancing Adversarial Robustness via Test-time Transformation Ensembling [51.51139269928358]
  We show how equipping models with Test-time Transformation Ensembling can work as a reliable defense against adversarial attacks. We show that TTE consistently improves model robustness against a variety of powerful attacks without any need for re-training.
  arXiv  Detail & Related papers  (2021-07-29T15:32:35Z)
• Policy Smoothing for Provably Robust Reinforcement Learning [109.90239627115336]
  We study the provable robustness of reinforcement learning against norm-bounded adversarial perturbations of the inputs. We generate certificates that guarantee that the total reward obtained by the smoothed policy will not fall below a certain threshold under a norm-bounded adversarial of perturbation the input.
  arXiv  Detail & Related papers  (2021-06-21T21:42:08Z)
• Boosting Adversarial Training with Hypersphere Embedding [53.75693100495097]
  Adversarial training is one of the most effective defenses against adversarial attacks for deep learning models. In this work, we advocate incorporating the hypersphere embedding mechanism into the AT procedure. We validate our methods under a wide range of adversarial attacks on the CIFAR-10 and ImageNet datasets.
  arXiv  Detail & Related papers  (2020-02-20T08:42:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.