NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving

• URL: http://arxiv.org/abs/2602.13293v1
• Date: Mon, 09 Feb 2026 05:42:59 GMT
• Title: NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving
• Abstract summary: Vision Language Models (VLMs) have advanced perception in autonomous driving (AD), but they remain vulnerable to adversarial threats.<n>We propose NutVLM, a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle.
• Score: 6.189595231250727
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models (VLMs) have advanced perception in autonomous driving (AD), but they remain vulnerable to adversarial threats. These risks range from localized physical patches to imperceptible global perturbations. Existing defense methods for VLMs remain limited and often fail to reconcile robustness with clean-sample performance. To bridge these gaps, we propose NutVLM, a comprehensive self-adaptive defense framework designed to secure the entire perception-decision lifecycle. Specifically, we first employ NutNet++ as a sentinel, which is a unified detection-purification mechanism. It identifies benign samples, local patches, and global perturbations through three-way classification. Subsequently, localized threats are purified via efficient grayscale masking, while global perturbations trigger Expert-guided Adversarial Prompt Tuning (EAPT). Instead of the costly parameter updates of full-model fine-tuning, EAPT generates "corrective driving prompts" via gradient-based latent optimization and discrete projection. These prompts refocus the VLM's attention without requiring exhaustive full-model retraining. Evaluated on the Dolphins benchmark, our NutVLM yields a 4.89% improvement in overall metrics (e.g., Accuracy, Language Score, and GPT Score). These results validate NutVLM as a scalable security solution for intelligent transportation. Our code is available at https://github.com/PXX/NutVLM.

Related papers

• Null-Space Constrained Low-Rank Adaptation for Response-Specified Large Language Model Unlearning [13.073474198841206]
  Large language model unlearning aims to suppress designated undesirable knowledge while preserving benign capabilities.<n>This paper introduces emphNull-Space Constrained Response-Specified Unlearning (NSRU)<n>NSRU uses an explicitly structured safe target response to specify the desired behavior for each forget query.
  arXiv  Detail & Related papers  (2026-06-09T15:26:36Z)
• EVA: Editing for Versatile Alignment against Jailbreaks [85.18411558566851]
  EVA (Editing for Versatile Alignment against Jailbreaks) is a novel framework that pioneers the application of direct model editing for safety alignment.<n> EVA identifies and surgically edits specific neurons responsible for the model's susceptibility to harmful instructions.<n>It effectively neutralizes harmful behaviors without compromising the model's general reasoning capabilities.
  arXiv  Detail & Related papers  (2026-05-14T12:16:10Z)
• GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimization [38.17733373188058]
  Diffusion Vision-Language Models (dVLMs) have demonstrated remarkable efficacy in multimodal tasks by departing from the traditional autoregressive generation paradigm.<n>To exploit this vulnerability, we propose Global Probability Optimization (GPO), a general jailbreak paradigm designed specifically for the denoising trajectory of masked diffusion models.<n>Building on this, we introduce GPO-V, the first visual-modality jailbreak framework tailored for dVLMs.
  arXiv  Detail & Related papers  (2026-05-08T07:54:57Z)
• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models [64.99746027349767]
  Adversarial Robustness Adaptation (R-Adapt) is a framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers.<n>R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness.
  arXiv  Detail & Related papers  (2026-03-13T09:02:11Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• NAAMSE: Framework for Evolutionary Security Evaluation of Agents [1.0131895986034316]
  We propose NAAMSE, an evolutionary framework that reframes agent security evaluation as a feedback-driven optimization problem.<n>Our system employs a single autonomous agent that orchestrates a lifecycle of genetic prompt mutation, hierarchical corpus exploration, and asymmetric behavioral scoring.<n>Experiments on Gemini 2.5 Flash demonstrate that evolutionary mutation systematically amplifies vulnerabilities missed by one-shot methods.
  arXiv  Detail & Related papers  (2026-02-07T06:13:02Z)
• Inject Once Survive Later: Backdooring Vision-Language-Action Models to Persist Through Downstream Fine-tuning [31.711305518359424]
  Vision-Language-Action (VLA) models have become foundational to modern embodied AI systems.<n>Recent methods attempt to inject backdoors into VLA models, but these backdoors are easily erased during downstream adaptation.<n>We propose INFUSE, the first backdoor attack framework for VLA base models that remains effective even with arbitrary user fine-tuning.
  arXiv  Detail & Related papers  (2026-01-31T03:59:07Z)
• The Compliance Paradox: Semantic-Instruction Decoupling in Automated Academic Code Evaluation [11.984098021215878]
  We introduce the Semantic-Preserving Adrial Code Injection (SPACI) Framework and the Abstract Syntax Tree-Aware Semantic Injection Protocol (AST-ASIP)<n>These methods exploit the Syntax-Semantics Gap by embedding adversarial directives into syntactically inert regions (trivia nodes) of the Abstract Syntax Tree.<n>Through a large-scale evaluation of 9 SOTA models across 25,000 submissions in Python, C, C++, and Java, we reveal catastrophic failure rates (>95%) in high-capacity open-weights models like DeepSeek-V3.
  arXiv  Detail & Related papers  (2026-01-29T07:40:58Z)
• State Backdoor: Towards Stealthy Real-world Poisoning Attack on Vision-Language-Action Model in State Space [42.234025453061875]
  Vision-Language-Action (VLA) models are widely deployed in safety-critical embodied AI applications such as robotics.<n>We introduce the State Backdoor, a novel and practical backdoor attack that leverages the robot arm's initial state as the trigger.<n>Our method achieves over 90% attack success rate without affecting benign task performance, revealing an underexplored vulnerability in embodied AI systems.
  arXiv  Detail & Related papers  (2026-01-07T08:54:31Z)
• V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs [66.81402538540458]
  We propose V-Attack, a novel method for precise local semantic attacks.<n>V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-11-25T11:51:17Z)
• AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models [60.39655329875822]
  Vision-Language-Action (VLA) models enable robots to interpret natural-language instructions and perform diverse tasks.<n>Despite growing interest in attacking such models, the effectiveness of existing techniques remains unclear.<n>We propose AttackVLA, a unified framework that aligns with the VLA development lifecycle.
  arXiv  Detail & Related papers  (2025-11-15T10:30:46Z)
• Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias [3.178301843099705]
  Large Language Models (LLMs) are increasingly trusted to perform automated code review and static analysis at scale.<n>This paper identifies and exploit a critical vulnerability in LLM-based code analysis.<n>We develop a fully automated, black-box algorithm that discovers and injects Familiar Pattern Attack (FPA) into target code.
  arXiv  Detail & Related papers  (2025-08-24T13:42:48Z)
• Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents [54.35629963816521]
  This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents.<n>The attack injects malicious behaviors into the model by modifying only the visual input.<n>We show that our attack achieves high success rates while preserving clean-task behavior.
  arXiv  Detail & Related papers  (2025-06-16T08:09:32Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves? [61.899791071654654]
  We investigate whether Vision-Language Models (VLMs) can improve their semantic grounding by "receiving" feedback.<n>We find that if prompted appropriately, VLMs can utilize feedback both in a single step and iteratively.<n>We show grounding accuracy consistently improves using automated feedback across all models in all settings investigated.
  arXiv  Detail & Related papers  (2024-04-09T17:59:04Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.