LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving

• URL: http://arxiv.org/abs/2512.24712v2
• Date: Sun, 04 Jan 2026 02:50:40 GMT
• Title: LSRE: Latent Semantic Rule Encoding for Real-Time Semantic Risk Detection in Autonomous Driving
• Abstract summary: Latent Semantic Rule framework converts sparsely sampled VLM judgments into decision boundaries within latent space of recurrent world model.<n>Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline.<n>LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.
• Score: 14.484350661315212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world autonomous driving must adhere to complex human social rules that extend beyond legally codified traffic regulations. Many of these semantic constraints, such as yielding to emergency vehicles, complying with traffic officers' gestures, or stopping for school buses, are intuitive for humans yet difficult to encode explicitly. Although large vision-language models (VLMs) can interpret such semantics, their inference cost makes them impractical for real-time deployment. This work proposes LSRE, a Latent Semantic Rule Encoding framework that converts sparsely sampled VLM judgments into decision boundaries within the latent space of a recurrent world model. By encoding language-defined safety semantics into a lightweight latent classifier, LSRE enables real-time semantic risk assessment at 10 Hz without per-frame VLM queries. Experiments on six semantic-failure scenarios in CARLA demonstrate that LSRE attains semantic risk detection accuracy comparable to a large VLM baseline, while providing substantially earlier hazard anticipation and maintaining low computational latency. LSRE further generalizes to rarely seen semantic-similar test cases, indicating that language-guided latent classification offers an effective and deployable mechanism for semantic safety monitoring in autonomous driving.

Related papers

• Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving [12.828441606176447]
  Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories.<n>We propose Lagrange, an open-vocabulary, computationally sparse driving framework based on Masked Latent Fields (MLF)<n>We introduce an intent-driven masked cross-attention module that temporally filters irrelevant entities, decoding the attended tokens into an implicit continuous energy field defined over spatial coordinates.
  arXiv  Detail & Related papers  (2026-06-18T14:18:01Z)
• GuardAD: Safeguarding Autonomous Driving MLLMs via Markovian Safety Logic [51.85782253444399]
  Multimodal large language models (MLLMs) are increasingly integrated into autonomous driving systems.<n>They remain vulnerable to diverse safety threats, particularly in accident-prone scenarios.<n>We propose GuardAD, a model-agnostic safeguard that formulates AD safety as an evolving Markovian logical state.
  arXiv  Detail & Related papers  (2026-05-11T11:28:03Z)
• Towards Lawful Autonomous Driving: Deriving Scenario-Aware Driving Requirements from Traffic Laws and Regulations [9.690981877708461]
  Driving in compliance with traffic laws and regulations is a basic requirement for human drivers, yet autonomous vehicles (AVs) can violate these requirements in diverse real-world scenarios.<n>We propose a novel pipeline that grounds reasoning in a traffic scenario taxonomy through node-wise anchors that encode hierarchical semantics.<n>Our method improves law-scenario matching by 29.1% and increases the accuracy of derived mandatory and prohibitive requirements by 36.9% and 38.2%, respectively.
  arXiv  Detail & Related papers  (2026-04-27T14:49:44Z)
• SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues [50.928203120795786]
  We study whether multimodal safety behavior in vision-language models can be steered by simple semantic cues.<n>We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content.
  arXiv  Detail & Related papers  (2026-03-19T16:18:00Z)
• DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Driving [7.788062051923755]
  DriveVLM-RL is a neuroscience-inspired framework that integrates vision-language models into reinforcement learning.<n>The framework decomposes semantic reward learning into a Static Pathway for continuous spatial safety assessment.<n>A hierarchical reward synthesis mechanism fuses semantic signals with vehicle states, while an asynchronous training pipeline decouples expensive VLM inference from environment interaction.
  arXiv  Detail & Related papers  (2026-03-18T21:55:29Z)
• SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving [52.02379432801349]
  We propose SGDrive, a novel framework that structures the VLM's representation learning around driving-specific knowledge hierarchies.<n>Built upon a pre-trained VLM backbone, SGDrive decomposes driving understanding into a scene-agent-goal hierarchy that mirrors human driving cognition.
  arXiv  Detail & Related papers  (2026-01-09T08:55:42Z)
• Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition [15.4994260281059]
  Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving.<n>We introduce TLS-Assist, a modular redundancy layer that augments LLM-based autonomous driving agents with explicit traffic light and sign recognition.<n>We demonstrate relative driving performance improvements of up to 14% over LMDrive and 7% over BEVDriver.
  arXiv  Detail & Related papers  (2025-11-18T11:52:52Z)
• BLIP-FusePPO: A Vision-Language Deep Reinforcement Learning Framework for Lane Keeping in Autonomous Vehicles [0.0]
  We propose a novel framework for multimodal reinforcement learning (RL) for autonomous lane-keeping (LK)<n>The proposed method lets the agent learn driving rules that are aware of their surroundings and easy to understand.<n>A hybrid reward function that includes semantic alignment, LK accuracy, obstacle avoidance, and speed regulation helps learning to be more efficient and generalizable.
  arXiv  Detail & Related papers  (2025-10-25T17:27:08Z)
• Discrete Diffusion for Reflective Vision-Language-Action Models in Autonomous Driving [55.13109926181247]
  We introduce ReflectDrive, a learning-based framework that integrates a reflection mechanism for safe trajectory generation via discrete diffusion.<n>Central to our approach is a safety-aware reflection mechanism that performs iterative self-correction without gradient.<n>Our method begins with goal-conditioned trajectory generation to model multi-modal driving behaviors.
  arXiv  Detail & Related papers  (2025-09-24T13:35:15Z)
• Automating Steering for Safe Multimodal Large Language Models [58.36932318051907]
  We introduce a modular and adaptive inference-time intervention technology, AutoSteer, without requiring any fine-tuning of the underlying model.<n>AutoSteer incorporates three core components: (1) a novel Safety Awareness Score (SAS) that automatically identifies the most safety-relevant distinctions among the model's internal layers; (2) an adaptive safety prober trained to estimate the likelihood of toxic outputs from intermediate representations; and (3) a lightweight Refusal Head that selectively intervenes to modulate generation when safety risks are detected.
  arXiv  Detail & Related papers  (2025-07-17T16:04:55Z)
• LD-Scene: LLM-Guided Diffusion for Controllable Generation of Adversarial Safety-Critical Driving Scenarios [3.6585028071015007]
  LD-Scene is a novel framework that integrates Large Language Models (LLMs) with Latent Diffusion Models (LDMs) for user-controllable adversarial scenario generation through natural language.<n>Our approach comprises an LDM that captures realistic driving distributions and an LLM-based guidance module that translates user queries into adversarial loss functions.<n>Our framework provides fine-grained control over adversarial behaviors, thereby facilitating more effective testing tailored to specific driving scenarios.
  arXiv  Detail & Related papers  (2025-05-16T13:41:05Z)
• Combating Partial Perception Deficit in Autonomous Driving with Multimodal LLM Commonsense [19.797977882386736]
  LLM-RCO is a framework to integrate human-like driving commonsense into autonomous systems facing perception deficits.<n>DriveLM-Deficit is a dataset of 53,895 video clips featuring deficits of safety-critical objects.<n>Our results show that LLMs fine-tuned with DriveLM-Deficit can enable more proactive movements instead of conservative stops in the context of perception deficits.
  arXiv  Detail & Related papers  (2025-03-10T08:01:41Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
  This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems. LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning. We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
  arXiv  Detail & Related papers  (2023-11-28T03:13:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.