A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving

• URL: http://arxiv.org/abs/2607.07103v1
• Date: Wed, 08 Jul 2026 07:39:21 GMT
• Title: A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving
• Abstract summary: K-Risk is a knowledge-augmented dataset that combines structured driving trajectories with large language model generated semantic annotations for safety-critical driving scenarios.<n>Using a unified risk-centric extraction pipeline, K-Risk curates 31,398 high-risk events, together with a 1,036-event extreme subset of near-collision cases.
• Score: 9.113712739147317
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safe autonomous driving requires both rapid responses to common high-risk events and deeper reasoning over rare, extreme long-tail scenarios in traffic safety. These scenarios are severely under-represented in naturalistic driving data, and existing trajectory and language-augmented datasets seldom provide high-risk event labels, semantic annotations, and verifiable safety signals. Here we present K-Risk, a knowledge-augmented dataset that combines structured driving trajectories with large language model generated semantic annotations for safety-critical driving scenarios. K-Risk integrates 20 human-driven and autonomous-vehicle trajectory datasets from Europe, China, and the United States, covering highways, urban freeways, intersections, and roundabouts. Using a unified risk-centric extraction pipeline, K-Risk curates 31,398 high-risk events, together with a 1,036-event extreme subset of near-collision cases. Each event is released as a synchronized trajectory, metadata, and language triplet containing structured scenario descriptions, abnormal-behavior notifications, and, for a representative subset, causal risk analyses and action recommendations validated through a closed-loop simulator with iterative reflection. By combining multi-dimensional risk annotations, interpretable language supervision, and verifiable decisions, K-Risk bridges structured traffic trajectories, semantic reasoning, and decision supervision, providing a standardized foundation for developing and evaluating next-generation risk-aware autonomous driving agents.

Related papers

• World Engine: Towards the Era of Post-Training for Autonomous Driving [63.61277741443647]
  We introduce World Engine, a generative framework that reconstructs high-fidelity interactive environments from real-world logs.<n>World Engine substantially reduces failures in rare safety-critical scenarios and yields significantly larger gains than scaling pre-training data alone.<n>When deployed on a production-scale autonomous driving system, the resulting policy reduces simulated collisions.
  arXiv  Detail & Related papers  (2026-06-18T06:28:33Z)
• Real-Time Driver Safety Scoring Through Inverse Crash Probability Modeling [0.0]
  This research introduces SafeDriver-IQ, a framework that transforms binary crash classifiers into continuous 0-100 safety scores.<n> Evaluation across 15 complementary analyses indicates that the framework reliably differentiates high-risk from low-risk driving conditions.<n>Findings reveal that 87% of crashes involve multiple co-occurring risk factors, with non-linear compounding effects that increase the risk to 4.5x baseline.
  arXiv  Detail & Related papers  (2026-03-16T05:37:12Z)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data [76.18834864749606]
  LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm.<n>Existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level.<n>We introduce AuraGen, a controllable engine that synthesizes benign trajectories, injects category-labeled risks with difficulty, and filters outputs via an automated reward model.
  arXiv  Detail & Related papers  (2025-10-10T18:42:32Z)
• CrashAgent: Crash Scenario Generation via Multi-modal Reasoning [34.42773413989066]
  We introduce CrashAgent, a framework designed to interpret multi-modal real-world traffic crash reports.<n>We evaluate the generated crash scenarios from multiple perspectives, including the accuracy of layout reconstruction, collision rate, and diversity.<n>The resulting high-quality and large-scale crash dataset will be publicly available to support the development of safe driving algorithms.
  arXiv  Detail & Related papers  (2025-05-23T19:55:32Z)
• Learning collision risk proactively from naturalistic driving data at scale [3.1457219084519004]
  This study introduces the Generalised Surrogate Safety Measure (GSSM)<n>GSSM learns collision risk from naturalistic driving without the need for crash or risk labels.<n> Diverse data from naturalistic driving, including motion kinematics, weather, lighting, etc., are used to train multiple GSSMs.<n>A basic GSSM using only instantaneous motion kinematics achieves an area under the precision-recall curve of 0.9 and secures a median time advance of 2.6 seconds to prevent potential collisions.
  arXiv  Detail & Related papers  (2025-05-19T07:22:32Z)
• Towards Reliable and Interpretable Traffic Crash Pattern Prediction and Safety Interventions Using Customized Large Language Models [14.53510262691888]
  TrafficSafe is a framework that adapts to reframe crash prediction and feature attribution as text-level reasoning.<n>Alcohol-impaired driving is the leading factor in severe crashes.<n>TrafficSafe highlights pivotal features during model training guiding strategic crash data collection improvements.
  arXiv  Detail & Related papers  (2025-05-18T21:02:30Z)
• RiskNet: Interaction-Aware Risk Forecasting for Autonomous Driving in Long-Tail Scenarios [6.024186631622774]
  RiskNet is an interaction-aware risk forecasting framework for autonomous vehicles.<n>It integrates deterministic risk modeling with probabilistic behavior prediction for comprehensive risk assessment.<n>It supports real-time, scenario-adaptive risk forecasting and demonstrates strong generalization across uncertain driving environments.
  arXiv  Detail & Related papers  (2025-04-22T02:36:54Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Traffic and Safety Rule Compliance of Humans in Diverse Driving Situations [48.924085579865334]
  Analyzing human data is crucial for developing autonomous systems that replicate safe driving practices. This paper presents a comparative evaluation of human compliance with traffic and safety rules across multiple trajectory prediction datasets.
  arXiv  Detail & Related papers  (2024-11-04T09:21:00Z)
• Learning Traffic Crashes as Language: Datasets, Benchmarks, and What-if Causal Analyses [76.59021017301127]
  We propose a large-scale traffic crash language dataset, named CrashEvent, summarizing 19,340 real-world crash reports. We further formulate the crash event feature learning as a novel text reasoning problem and further fine-tune various large language models (LLMs) to predict detailed accident outcomes. Our experiments results show that our LLM-based approach not only predicts the severity of accidents but also classifies different types of accidents and predicts injury outcomes.
  arXiv  Detail & Related papers  (2024-06-16T03:10:16Z)
• DRAMA: Joint Risk Localization and Captioning in Driving [23.091433343825727]
  This paper proposes a new research direction of joint risk localization in driving scenes and its risk explanation as a natural language description. Due to the lack of standard benchmarks, we collected a large-scale dataset, DRAMA (Driving Risk Assessment Mechanism with A captioning module) Our dataset accommodates video- and object-level questions on driving risks with associated important objects to achieve the goal of visual captioning.
  arXiv  Detail & Related papers  (2022-09-22T03:53:56Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.