Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles

• URL: http://arxiv.org/abs/2608.08815v1
• Date: Sun, 09 Aug 2026 17:05:00 GMT
• Title: Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles
• Abstract summary: LAMDA is a training framework that transfers language-grounded structure into TSR models without using adversarial examples or adding inference-time overhead.<n>It consistently improves robustness across all attack-backbone-dataset combinations, with gains of up to +12.5 pp under shadow attacks and +13.2 pp under natural-light attacks.
• Score: 1.6385815610837167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Traffic sign recognition (TSR) models based on deep neural networks achieve strong clean-data performance but remain vulnerable to physically realizable adversarial attacks, including shadow perturbations, natural-light interference, and printed patches. Existing defenses often improve robustness against one attack type while degrading performance on others, and can reduce clean accuracy. We propose LAMDA (Language-Anchored Model for Direction Alignment), a training framework that transfers language-grounded structure into TSR models without using adversarial examples or adding inference-time overhead. LAMDA builds two fixed prototype banks from VLM-generated sign descriptions and class names using a frozen OpenCLIP text encoder, and uses them to supervise visual features through two complementary auxiliary losses during training. At inference, the adapter and prototype banks are discarded, leaving a standard backbone and classifier. Evaluated on GTSRB and LISA across four backbones and three physical attack types, LAMDA is the only method among ten evaluated that consistently improves robustness across all attack-backbone-dataset combinations, with gains of up to +12.5 pp under shadow attacks and +13.2 pp under natural-light attacks, while preserving or improving clean accuracy in nearly all cases.

Related papers

• Towards Understanding the Robustness of Sparse Autoencoders [13.16745936025085]
  We present a study of integrating pretrained SAEs into transformer residual streams at inference time.<n>SAE-augmented models achieve up to a 5x reduction in jailbreak success rate relative to the undefended baseline.
  arXiv  Detail & Related papers  (2026-04-20T19:00:09Z)
• ProtoGuard-SL: Prototype Consistency Based Backdoor Defense for Vertical Split Learning [4.085737586606009]
  We propose ProtoGuard-SL, a server-side defense that improves the robustness of split learning by exploiting class-conditional representation consistency in the embedding space.<n>Our approach is motivated by the observation that benign embeddings within the same class exhibit stable semantic alignment, whereas poisoned embeddings inevitably disrupt this structure.
  arXiv  Detail & Related papers  (2026-04-04T05:24:09Z)
• What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models [64.99746027349767]
  Adversarial Robustness Adaptation (R-Adapt) is a framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers.<n>R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness.
  arXiv  Detail & Related papers  (2026-03-13T09:02:11Z)
• Transferable Backdoor Attacks for Code Models via Sharpness-Aware Adversarial Perturbation [37.091275561451695]
  Existing backdoor attacks on code models face a fundamental trade-off between transferability and stealthiness.<n>We propose Sharpness-aware Transferable Adversarial Backdoor (STAB)<n>STAB achieves both transferability and stealthiness without requiring complete victim data.
  arXiv  Detail & Related papers  (2026-02-11T08:26:47Z)
• Robust Anti-Backdoor Instruction Tuning in LVLMs [53.766434746801366]
  We introduce a lightweight, certified-agnostic defense framework for large visual language models (LVLMs)<n>Our framework finetunes only adapter modules and text embedding layers under instruction tuning.<n>Experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero.
  arXiv  Detail & Related papers  (2025-06-04T01:23:35Z)
• Robustifying Vision-Language Models via Dynamic Token Reweighting [28.675118345987887]
  Large vision-language models (VLMs) are highly vulnerable to jailbreak attacks.<n>We present a novel inference-time defense that mitigates multimodal jailbreak attacks.<n>We introduce a new formulation of the safety-relevant distributional shift induced by the visual modality.
  arXiv  Detail & Related papers  (2025-05-22T03:00:39Z)
• Deep Positive-Negative Prototypes for Adversarially Robust Discriminative Prototypical Learning [0.24999074238880484]
  We propose a novel framework named Adversarially trained Deep Positive-Negative Prototypes (Adv-DPNP)<n>Adv-DPNP integrates discriminative prototype-based learning with adversarial training.<n>We show that Adv-DPNP achieves the highest average accuracy across severities and corruption types.
  arXiv  Detail & Related papers  (2025-04-03T15:42:58Z)
• Overlap-Aware Feature Learning for Robust Unsupervised Domain Adaptation for 3D Semantic Segmentation [9.578322021478426]
  3D point cloud semantic segmentation (PCSS) is a cornerstone for environmental perception in robotic systems and autonomous driving.<n>Existing methods critically overlook the inherent vulnerability to real-world perturbations (e.g., snow, fog, rain) and adversarial distortions.<n>This work first identifies two intrinsic limitations that undermine current PCSS-UDA robustness.<n>We propose a tripartite framework consisting of: 1) a robustness evaluation model quantifying resilience against adversarial attack/corruption types through robustness metrics; 2) an invertible attention alignment module (IAAM) enabling bidirectional domain mapping while preserving discriminative structure via attention-guided overlap suppression; and
  arXiv  Detail & Related papers  (2025-04-02T12:16:23Z)
• Efficient Adversarial Training in LLMs with Continuous Attacks [99.5882845458567]
  Large language models (LLMs) are vulnerable to adversarial attacks that can bypass their safety guardrails. We propose a fast adversarial training algorithm (C-AdvUL) composed of two losses. C-AdvIPO is an adversarial variant of IPO that does not require utility data for adversarially robust alignment.
  arXiv  Detail & Related papers  (2024-05-24T14:20:09Z)
• FACTUAL: A Novel Framework for Contrastive Learning Based Robust SAR Image Classification [10.911464455072391]
  FACTUAL is a Contrastive Learning framework for Adversarial Training and robust SAR classification. Our model achieves 99.7% accuracy on clean samples, and 89.6% on perturbed samples, both outperforming previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-04-04T06:20:22Z)
• Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization [53.04697800214848]
  Adversarial Training (AT) is pivotal in fortifying the robustness of deep learning models. AT methods, relying on direct iterative updates for target model's defense, frequently encounter obstacles such as unstable training and catastrophic overfitting. We present a general proxy guided defense framework, LAST' (bf Learn from the Pbf ast)
  arXiv  Detail & Related papers  (2023-10-19T13:13:41Z)
• PointACL:Adversarial Contrastive Learning for Robust Point Clouds Representation under Adversarial Attack [73.3371797787823]
  Adversarial contrastive learning (ACL) is considered an effective way to improve the robustness of pre-trained models. We present our robust aware loss function to train self-supervised contrastive learning framework adversarially. We validate our method, PointACL on downstream tasks, including 3D classification and 3D segmentation with multiple datasets.
  arXiv  Detail & Related papers  (2022-09-14T22:58:31Z)
• Discriminator-Free Generative Adversarial Attack [87.71852388383242]
  Agenerative-based adversarial attacks can get rid of this limitation. ASymmetric Saliency-based Auto-Encoder (SSAE) generates the perturbations. The adversarial examples generated by SSAE not only make thewidely-used models collapse, but also achieves good visual quality.
  arXiv  Detail & Related papers  (2021-07-20T01:55:21Z)
• A Self-supervised Approach for Adversarial Robustness [105.88250594033053]
  Adversarial examples can cause catastrophic mistakes in Deep Neural Network (DNNs) based vision systems. This paper proposes a self-supervised adversarial training mechanism in the input space. It provides significant robustness against the textbfunseen adversarial attacks.
  arXiv  Detail & Related papers  (2020-06-08T20:42:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.