Why Braking? Scenario Extraction and Reasoning Utilizing LLM

• URL: http://arxiv.org/abs/2507.15874v1
• Date: Thu, 17 Jul 2025 08:33:56 GMT
• Title: Why Braking? Scenario Extraction and Reasoning Utilizing LLM
• Authors: Yin Wu, Daniel Slieter, Vivek Subramanian, Ahmed Abouelazm, Robin Bohn, J. Marius Zöllner,
• Abstract summary: We propose a novel framework that leverages Large Language Model (LLM) for scenario understanding and reasoning.<n>Our method bridges the gap between low-level numerical signals and natural language descriptions, enabling LLM to interpret and classify driving scenarios.
• Score: 13.88343221678386
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing number of ADAS-equipped vehicles has led to a dramatic increase in driving data, yet most of them capture routine driving behavior. Identifying and understanding safety-critical corner cases within this vast dataset remains a significant challenge. Braking events are particularly indicative of potentially hazardous situations, motivating the central question of our research: Why does a vehicle brake? Existing approaches primarily rely on rule-based heuristics to retrieve target scenarios using predefined condition filters. While effective in simple environments such as highways, these methods lack generalization in complex urban settings. In this paper, we propose a novel framework that leverages Large Language Model (LLM) for scenario understanding and reasoning. Our method bridges the gap between low-level numerical signals and natural language descriptions, enabling LLM to interpret and classify driving scenarios. We propose a dual-path scenario retrieval that supports both category-based search for known scenarios and embedding-based retrieval for unknown Out-of-Distribution (OOD) scenarios. To facilitate evaluation, we curate scenario annotations on the Argoverse 2 Sensor Dataset. Experimental results show that our method outperforms rule-based baselines and generalizes well to OOD scenarios.

Related papers

• DRAMA-X: A Fine-grained Intent Prediction and Risk Reasoning Benchmark For Driving [5.362063089413001]
  No existing benchmark evaluates multi-class intent prediction in safety-critical situations.<n>We introduce DRAMA-X, a fine-grained benchmark constructed from the DRAMA dataset.<n>We propose SGG-Intent, a lightweight, training-free framework that mirrors the ego vehicle's reasoning pipeline.
  arXiv  Detail & Related papers  (2025-06-21T05:01:42Z)
• From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios [6.681744368557208]
  Large Language Models (LLMs) and structured scenario parsing and prompt engineering are used to generate safety-critical driving scenarios.<n>We validate our approach using a 2D simulation framework and multiple pre-trained LLMs.<n>We conclude that an LLM equipped with domain-informed prompting techniques can effectively evaluate and generate safety-critical driving scenarios.
  arXiv  Detail & Related papers  (2025-02-04T09:19:13Z)
• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems.<n>We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics.<n>We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• Generating Out-Of-Distribution Scenarios Using Language Models [58.47597351184034]
  Large Language Models (LLMs) have shown promise in autonomous driving. This paper introduces a framework for generating diverse Out-Of-Distribution (OOD) driving scenarios. We evaluate our framework through extensive simulations and introduce a new "OOD-ness" metric.
  arXiv  Detail & Related papers  (2024-11-25T16:38:17Z)
• ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles [17.396416459648755]
  ChatScene is a Large Language Model (LLM)-based agent that generates safety-critical scenarios for autonomous vehicles. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets.
  arXiv  Detail & Related papers  (2024-05-22T23:21:15Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Realistic Safety-critical Scenarios Search for Autonomous Driving System via Behavior Tree [8.286351881735191]
  We propose the Matrix-Fuzzer, a behavior tree-based testing framework, to automatically generate realistic safety-critical test scenarios. Our approach is able to find the most types of safety-critical scenarios, but only generating around 30% of the total scenarios compared with the baseline algorithm.
  arXiv  Detail & Related papers  (2023-05-11T06:53:03Z)
• Behavioral Intention Prediction in Driving Scenes: A Survey [70.53285924851767]
  Behavioral Intention Prediction (BIP) simulates a human consideration process and fulfills the early prediction of specific behaviors. This work provides a comprehensive review of BIP from the available datasets, key factors and challenges, pedestrian-centric and vehicle-centric BIP approaches, and BIP-aware applications.
  arXiv  Detail & Related papers  (2022-11-01T11:07:37Z)
• Toward Unsupervised Test Scenario Extraction for Automated Driving Systems from Urban Naturalistic Road Traffic Data [0.0]
  The presented method deploys an unsupervised machine learning pipeline to extract scenarios from road traffic data. It is evaluated for naturalistic road traffic data at urban intersections from the inD and the Silicon Valley Intersections datasets. Using hierarchical clustering the results show both a jump in overall accuracy of around 20% when moving from 4 to 5 clusters and a saturation effect starting at 41 clusters with an overall accuracy of 84%.
  arXiv  Detail & Related papers  (2022-02-14T10:55:14Z)
• Divide-and-Conquer for Lane-Aware Diverse Trajectory Prediction [71.97877759413272]
  Trajectory prediction is a safety-critical tool for autonomous vehicles to plan and execute actions. Recent methods have achieved strong performances using Multi-Choice Learning objectives like winner-takes-all (WTA) or best-of-many. Our work addresses two key challenges in trajectory prediction, learning outputs, and better predictions by imposing constraints using driving knowledge.
  arXiv  Detail & Related papers  (2021-04-16T17:58:56Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts? [104.04999499189402]
  Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment. We propose an uncertainty-aware planning method, called emphrobust imitative planning (RIP) Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. We introduce an autonomous car novel-scene benchmark, textttCARNOVEL, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
  arXiv  Detail & Related papers  (2020-06-26T11:07:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.