Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition

• URL: http://arxiv.org/abs/2511.14391v1
• Date: Tue, 18 Nov 2025 11:52:52 GMT
• Title: Enhancing LLM-based Autonomous Driving with Modular Traffic Light and Sign Recognition
• Authors: Fabian Schmidt, Noushiq Mohammed Kayilan Abdul Nazar, Markus Enzweiler, Abhinav Valada,
• Abstract summary: Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving.<n>We introduce TLS-Assist, a modular redundancy layer that augments LLM-based autonomous driving agents with explicit traffic light and sign recognition.<n>We demonstrate relative driving performance improvements of up to 14% over LMDrive and 7% over BEVDriver.
• Score: 15.4994260281059
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) are increasingly used for decision-making and planning in autonomous driving, showing promising reasoning capabilities and potential to generalize across diverse traffic situations. However, current LLM-based driving agents lack explicit mechanisms to enforce traffic rules and often struggle to reliably detect small, safety-critical objects such as traffic lights and signs. To address this limitation, we introduce TLS-Assist, a modular redundancy layer that augments LLM-based autonomous driving agents with explicit traffic light and sign recognition. TLS-Assist converts detections into structured natural language messages that are injected into the LLM input, enforcing explicit attention to safety-critical cues. The framework is plug-and-play, model-agnostic, and supports both single-view and multi-view camera setups. We evaluate TLS-Assist in a closed-loop setup on the LangAuto benchmark in CARLA. The results demonstrate relative driving performance improvements of up to 14% over LMDrive and 7% over BEVDriver, while consistently reducing traffic light and sign infractions. We publicly release the code and models on https://github.com/iis-esslingen/TLS-Assist.

Related papers

• Virtual Traffic Police: Large Language Model-Augmented Traffic Signal Control for Unforeseen Incidents [5.077053934708947]
  We propose a hierarchical framework that augments existing traffic signal control systems with Large Language Models (LLMs)<n>A virtual traffic police agent at the upper level dynamically fine-tunes selected parameters of signal controllers at the lower level in response to real-time traffic incidents.<n>Our results show that LLMs can serve as trustworthy virtual traffic police officers that can adapt conventional TSC methods to unforeseen traffic incidents.
  arXiv  Detail & Related papers  (2026-01-22T10:04:21Z)
• VLDrive: Vision-Augmented Lightweight MLLMs for Efficient Language-grounded Autonomous Driving [90.21844353859454]
  We introduce a novel approach featuring a lightweight MLLM architecture with enhanced vision components.<n>VLDrive achieves state-of-the-art driving performance while reducing parameters by 81%.
  arXiv  Detail & Related papers  (2025-11-09T07:14:53Z)
• AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving [71.55254573283793]
  Existing approaches either activate Large Language Models too frequently, causing excessive computational overhead, or use fixed schedules.<n>We propose AdaDrive, an adaptively collaborative slow-fast framework that optimally determines when and how LLMs contribute to decision-making.<n>AdaDrive provides a flexible, context-aware framework that maximizes decision accuracy without compromising real-time performance.
  arXiv  Detail & Related papers  (2025-11-09T07:05:03Z)
• VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture [10.403439446065033]
  VLMLight is a novel framework that integrates vision-language meta-control with dual-branch reasoning.<n>A large language model (LLM) serves as a safety-prioritized meta-controller, selecting between a fast RL policy for routine traffic and a structured reasoning branch for critical cases.<n> Experiments show that VLMLight reduces waiting times for emergency vehicles by up to 65% over RL-only systems.
  arXiv  Detail & Related papers  (2025-05-26T04:12:57Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• TeLL-Drive: Enhancing Autonomous Driving with Teacher LLM-Guided Deep Reinforcement Learning [61.33599727106222]
  TeLL-Drive is a hybrid framework that integrates a Teacher LLM to guide an attention-based Student DRL policy.<n>A self-attention mechanism then fuses these strategies with the DRL agent's exploration, accelerating policy convergence and boosting robustness.
  arXiv  Detail & Related papers  (2025-02-03T14:22:03Z)
• Large Language Models (LLMs) as Traffic Control Systems at Urban Intersections: A New Paradigm [5.233512464561313]
  This study introduces a novel approach for traffic control systems by using Large Language Models (LLMs) as traffic controllers. The study utilizes their logical reasoning, scene understanding, and decision-making capabilities to optimize throughput and provide feedback based on traffic conditions in real-time.
  arXiv  Detail & Related papers  (2024-11-16T19:23:52Z)
• Strada-LLM: Graph LLM for traffic prediction [62.2015839597764]
  A considerable challenge in traffic prediction lies in handling the diverse data distributions caused by vastly different traffic conditions.<n>We propose a graph-aware LLM for traffic prediction that considers proximal traffic information.<n>We adopt a lightweight approach for efficient domain adaptation when facing new data distributions in few-shot fashion.
  arXiv  Detail & Related papers  (2024-10-28T09:19:29Z)
• A Superalignment Framework in Autonomous Driving with Large Language Models [2.650382010271]
  Large language models (LLMs) and multi-modal large language models (MLLMs) are extensively used in autonomous driving. Despite their importance, the security aspect of LLMs in autonomous driving remains underexplored. This research introduces a novel security framework for autonomous vehicles, utilizing a multi-agent LLM approach.
  arXiv  Detail & Related papers  (2024-06-09T05:26:38Z)
• LLMLight: Large Language Models as Traffic Signal Control Agents [25.438040499152745]
  Traffic Signal Control (TSC) is a crucial component in urban traffic management, aiming to optimize road network efficiency and reduce congestion.<n>This paper presents LLMLight, a novel framework employing Large Language Models (LLMs) as decision-making agents for TSC.
  arXiv  Detail & Related papers  (2023-12-26T13:17:06Z)
• LanguageMPC: Large Language Models as Decision Makers for Autonomous Driving [84.31119464141631]
  This work employs Large Language Models (LLMs) as a decision-making component for complex autonomous driving scenarios.<n>Extensive experiments demonstrate that our proposed method not only consistently surpasses baseline approaches in single-vehicle tasks, but also helps handle complex driving behaviors even multi-vehicle coordination.
  arXiv  Detail & Related papers  (2023-10-04T17:59:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.