VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving

• URL: http://arxiv.org/abs/2607.05180v1
• Date: Mon, 06 Jul 2026 14:58:47 GMT
• Title: VLM-CASE: Vision-Language Model Enabled Context-Adaptive Safety Envelopes for Anticipatory Safe Autonomous Driving
• Abstract summary: Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade what the vehicle can perceive and what it can physically do.<n>This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times.<n>We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions.
• Score: 9.642572690101325
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adverse driving conditions, such as bad weather, remain a principal barrier to autonomous driving because they degrade two things at once: what the vehicle can perceive and what it can physically do. Human drivers cope by anticipation, reasoning about the scene and re-budgeting speed, following distance, and steering before grip or sight is lost, whereas current autonomous driving systems at best react after the fact. This paper proposes VLM-CASE, a framework that gives an autonomous vehicle this anticipatory capacity while keeping its motion bounded by a formal safety model at all times. A vision-language model (VLM), fine-tuned with low-rank adaptation (LoRA), reasons about the scene from the front-camera image and reports the road surface and visibility conditions. This output parametrizes a context-adaptive safety envelope (CASE), derived from physical limits and the guarantees of responsibility-sensitive safety, that couples braking and steering through a shared friction budget. A model predictive controller then drives freely within the envelope, while the VLM runs asynchronously so it never blocks the real-time control loop. We validate the framework in closed-loop CARLA simulation on tasks that demand both lateral and longitudinal control, across a range of weather, road-surface, and lighting conditions. The resulting controller, VLM-CASE-MPC, completes all trials, outperforming a conventional MPC baseline and a state-of-the-art VLM-integrated controller. Ablations confirm that the gains come from context adaptation, with the friction and visibility adaptations proving complementary. Furthermore, the framework is controller-agnostic and pairs with almost any low-level controller, offering a promising direction for safe autonomous driving. The dataset and supplementary materials for VLM-CASE are available at https://github.com/ytj254/VLM-CASE.

Related papers

• Driving Context-guided Model Predictive Planning and Control for Autonomous Car Racing at the Limit and Beyond [2.1824191810542666]
  This paper presents a Model Predictive Control-based motion planning and control pipeline for autonomous car racing.<n>It is validated on the fully autonomous open-wheel racecar Superformula EAV-25, with a lap time that is within 2% of the best human driver reference.
  arXiv  Detail & Related papers  (2026-09-11T10:06:11Z)
• End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation [59.188006170485046]
  E2E-CDiff is an end-to-end conditional diffusion framework for controllable and realistic scenario generation.<n>Differentiable guidance regulates speed, enforces drivable-area compliance, and supports collision-avoidance or collision-seeking behaviors.<n>E2E-CDiff also performs competitively as a learning-based ego planner.
  arXiv  Detail & Related papers  (2026-07-21T02:13:25Z)
• An LLM-Explainable DRL Framework for Passenger-Directed Autonomous Driving [2.0646127669654826]
  This work combines deep reinforcement learning (DRL) for adaptive driving control with large language model (LLM)-based explainability modules.<n>DRL agents were trained in simulation using a Dueling Double Deep Q-Network to follow distinct driving requests.<n>LLM modules were introduced to interpret passenger requests, determine when explanations were needed, and generate concise, safety-oriented justifications.
  arXiv  Detail & Related papers  (2026-06-02T15:15:05Z)
• Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout [55.19244743715421]
  Driver-WM is a driver-centric latent world model that rolls out in-cabin dynamics causally conditioned on out-cabin traffic context.<n>It unifies physical kinematics forecasting with auxiliary behavioral and emotional semantic recognition.<n>Driver-WM yields robust long-horizon geometric forecasting for reactive high-motion maneuvers.
  arXiv  Detail & Related papers  (2026-05-06T16:30:48Z)
• DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving [7.788062051923755]
  DriveVLM-RL is a neuroscience-inspired framework that integrates vision-language models into reinforcement learning.<n>The framework decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment.<n>A hierarchical reward synthesis mechanism fuses semantic signals with vehicle states, while an asynchronous training pipeline decouples expensive VLM inference from environment interaction.
  arXiv  Detail & Related papers  (2026-03-18T21:55:29Z)
• VLM-AutoDrive: Post-Training Vision-Language Models for Safety-Critical Autonomous Driving Events [25.118024547769895]
  We introduce VLM-AutoDrive, a modular framework for adapting pretrained Vision-Language Models to high-fidelity anomaly detection.<n>Off-the-shelf VLMs such as NVIDIA's Cosmos-1 7B (CR1) exhibit near-zero Collision recall in zero-shot settings.<n>VLM-AutoDrive offers a scalable recipe for adapting general-purpose VLMs to safety-critical, temporally localized perception tasks.
  arXiv  Detail & Related papers  (2026-03-18T18:23:34Z)
• LMAD: Integrated End-to-End Vision-Language Model for Explainable Autonomous Driving [58.535516533697425]
  Large vision-language models (VLMs) have shown promising capabilities in scene understanding.<n>We propose a novel vision-language framework tailored for autonomous driving, called LMAD.<n>Our framework emulates modern end-to-end driving paradigms by incorporating comprehensive scene understanding and a task-specialized structure with VLMs.
  arXiv  Detail & Related papers  (2025-08-17T15:42:54Z)
• VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture [10.403439446065033]
  VLMLight is a novel framework that integrates vision-language meta-control with dual-branch reasoning.<n>A large language model (LLM) serves as a safety-prioritized meta-controller, selecting between a fast RL policy for routine traffic and a structured reasoning branch for critical cases.<n> Experiments show that VLMLight reduces waiting times for emergency vehicles by up to 65% over RL-only systems.
  arXiv  Detail & Related papers  (2025-05-26T04:12:57Z)
• Natural Reflection Backdoor Attack on Vision Language Model for Autonomous Driving [55.96227460521096]
  Vision-Language Models (VLMs) have been integrated into autonomous driving systems to enhance reasoning capabilities.<n>We propose a natural reflection-based backdoor attack targeting VLM systems in autonomous driving scenarios.<n>Our findings uncover a new class of attacks that exploit the stringent real-time requirements of autonomous driving.
  arXiv  Detail & Related papers  (2025-05-09T20:28:17Z)
• Combating Partial Perception Deficit in Autonomous Driving with Multimodal LLM Commonsense [19.797977882386736]
  LLM-RCO is a framework to integrate human-like driving commonsense into autonomous systems facing perception deficits.<n>DriveLM-Deficit is a dataset of 53,895 video clips featuring deficits of safety-critical objects.<n>Our results show that LLMs fine-tuned with DriveLM-Deficit can enable more proactive movements instead of conservative stops in the context of perception deficits.
  arXiv  Detail & Related papers  (2025-03-10T08:01:41Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Differentiable Control Barrier Functions for Vision-based End-to-End Autonomous Driving [100.57791628642624]
  We introduce a safety guaranteed learning framework for vision-based end-to-end autonomous driving. We design a learning system equipped with differentiable control barrier functions (dCBFs) that is trained end-to-end by gradient descent.
  arXiv  Detail & Related papers  (2022-03-04T16:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.