SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation

• URL: http://arxiv.org/abs/2505.09427v2
• Date: Thu, 15 May 2025 07:22:20 GMT
• Title: SafePath: Conformal Prediction for Safe LLM-Based Autonomous Navigation
• Authors: Achref Doula, Max Mühlhäuser, Alejandro Sanchez Guinea,
• Abstract summary: We introduce SafePath, a framework that augments Large Language Models (LLMs) with formal safety guarantees.<n>In the first stage, we use an LLM that generates a set of diverse candidate paths, exploring possible trajectories based on agent behaviors and environmental cues.<n>In the second stage, SafePath filters out high-risk trajectories while guaranteeing at least one safe option is included with a user-defined probability.<n>In the final stage, our approach selects the path with the lowest expected collision risk when uncertainty is low or delegates control to a human when uncertainty is high.
• Score: 67.22657932549723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) show growing promise in autonomous driving by reasoning over complex traffic scenarios to generate path plans. However, their tendencies toward overconfidence, and hallucinations raise critical safety concerns. We introduce SafePath, a modular framework that augments LLM-based path planning with formal safety guarantees using conformal prediction. SafePath operates in three stages. In the first stage, we use an LLM that generates a set of diverse candidate paths, exploring possible trajectories based on agent behaviors and environmental cues. In the second stage, SafePath filters out high-risk trajectories while guaranteeing that at least one safe option is included with a user-defined probability, through a multiple-choice question-answering formulation that integrates conformal prediction. In the final stage, our approach selects the path with the lowest expected collision risk when uncertainty is low or delegates control to a human when uncertainty is high. We theoretically prove that SafePath guarantees a safe trajectory with a user-defined probability, and we show how its human delegation rate can be tuned to balance autonomy and safety. Extensive experiments on nuScenes and Highway-env show that SafePath reduces planning uncertainty by 77\% and collision rates by up to 70\%, demonstrating effectiveness in making LLM-driven path planning more safer.

Related papers

• Pro2Guard: Proactive Runtime Enforcement of LLM Agent Safety via Probabilistic Model Checking [8.970702398918924]
  Large Language Model (LLM) agents exhibit powerful autonomous capabilities across domains such as robotics, virtual assistants, and web automation.<n>Existing rule-based enforcement systems, such as AgentSpec, focus on developing reactive safety rules.<n>We propose Pro2Guard, a proactive runtime enforcement framework grounded in probabilistic reachability analysis.
  arXiv  Detail & Related papers  (2025-08-01T10:24:47Z)
• DriveSuprim: Towards Precise Trajectory Selection for End-to-End Planning [43.284391163049236]
  DriveSuprim is a selection-based paradigm for trajectory selection in autonomous vehicles.<n>It achieves state-of-the-art performance, including collision avoidance and compliance with rules.<n>It maintains high trajectory quality in various driving scenarios.
  arXiv  Detail & Related papers  (2025-06-07T04:39:06Z)
• SafetyAnalyst: Interpretable, transparent, and steerable safety moderation for AI behavior [56.10557932893919]
  We present SafetyAnalyst, a novel AI safety moderation framework.<n>Given an AI behavior, SafetyAnalyst uses chain-of-thought reasoning to analyze its potential consequences.<n>It aggregates all harmful and beneficial effects into a harmfulness score using fully interpretable weight parameters.
  arXiv  Detail & Related papers  (2024-10-22T03:38:37Z)
• Automated and Complete Generation of Traffic Scenarios at Road Junctions Using a Multi-level Danger Definition [2.5608506499175094]
  We propose an approach to derive a complete set of (potentially dangerous) abstract scenarios at any given road junction.<n>From these abstract scenarios, we derive exact paths that actors must follow to guide simulation-based testing.<n>Results show that the AV-under-test is involved in increasing percentages of unsafe behaviors in simulation.
  arXiv  Detail & Related papers  (2024-10-09T17:23:51Z)
• Safe Policy Exploration Improvement via Subgoals [44.07721205323709]
  Reinforcement learning is a widely used approach to autonomous navigation, showing potential in various tasks and robotic setups. One of the main reasons for poor performance in such setups is that the need to respect the safety constraints degrades the exploration capabilities of an RL agent. We introduce a novel learnable algorithm that is based on decomposing the initial problem into smaller sub-problems via intermediate goals.
  arXiv  Detail & Related papers  (2024-08-25T16:12:49Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
  We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position.<n>DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful response sequence.
  arXiv  Detail & Related papers  (2024-07-12T09:36:33Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• Optimizing Trajectories for Highway Driving with Offline Reinforcement Learning [11.970409518725491]
  We propose a Reinforcement Learning-based approach to autonomous driving. We compare the performance of our agent against four other highway driving agents. We demonstrate that our offline trained agent, with randomly collected data, learns to drive smoothly, achieving as close as possible to the desired velocity, while outperforming the other agents.
  arXiv  Detail & Related papers  (2022-03-21T13:13:08Z)
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts? [104.04999499189402]
  Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment. We propose an uncertainty-aware planning method, called emphrobust imitative planning (RIP) Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. We introduce an autonomous car novel-scene benchmark, textttCARNOVEL, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
  arXiv  Detail & Related papers  (2020-06-26T11:07:32Z)
• Risk-Aware High-level Decisions for Automated Driving at Occluded Intersections with Reinforcement Learning [16.69903761648675]
  We propose a generic risk-aware DQN approach to learn high level actions for driving through unsignalized intersections. The proposed state representation provides lane based information which allows to be used for multi-lane scenarios. We also propose a risk based reward function which punishes risky situations instead of only collision failures.
  arXiv  Detail & Related papers  (2020-04-09T09:44:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.