DRARL: Disengagement-Reason-Augmented Reinforcement Learning for Efficient Improvement of Autonomous Driving Policy

• URL: http://arxiv.org/abs/2506.16720v1
• Date: Fri, 20 Jun 2025 03:32:01 GMT
• Title: DRARL: Disengagement-Reason-Augmented Reinforcement Learning for Efficient Improvement of Autonomous Driving Policy
• Authors: Weitao Zhou, Bo Zhang, Zhong Cao, Xiang Li, Qian Cheng, Chunyang Liu, Yaqin Zhang, Diange Yang,
• Abstract summary: disengagement-reason-augmented reinforcement learning (DRARL) enhances driving policy improvement process.<n>The method is evaluated using real-world disengagement cases collected by autonomous driving robotaxi.
• Score: 24.36567420971839
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the increasing presence of automated vehicles on open roads under driver supervision, disengagement cases are becoming more prevalent. While some data-driven planning systems attempt to directly utilize these disengagement cases for policy improvement, the inherent scarcity of disengagement data (often occurring as a single instances) restricts training effectiveness. Furthermore, some disengagement data should be excluded since the disengagement may not always come from the failure of driving policies, e.g. the driver may casually intervene for a while. To this end, this work proposes disengagement-reason-augmented reinforcement learning (DRARL), which enhances driving policy improvement process according to the reason of disengagement cases. Specifically, the reason of disengagement is identified by a out-of-distribution (OOD) state estimation model. When the reason doesn't exist, the case will be identified as a casual disengagement case, which doesn't require additional policy adjustment. Otherwise, the policy can be updated under a reason-augmented imagination environment, improving the policy performance of disengagement cases with similar reasons. The method is evaluated using real-world disengagement cases collected by autonomous driving robotaxi. Experimental results demonstrate that the method accurately identifies policy-related disengagement reasons, allowing the agent to handle both original and semantically similar cases through reason-augmented training. Furthermore, the approach prevents the agent from becoming overly conservative after policy adjustments. Overall, this work provides an efficient way to improve driving policy performance with disengagement cases.

Related papers

• RACER: Epistemic Risk-Sensitive RL Enables Fast Driving with Fewer Crashes [57.319845580050924]
  We propose a reinforcement learning framework that combines risk-sensitive control with an adaptive action space curriculum. We show that our algorithm is capable of learning high-speed policies for a real-world off-road driving task.
  arXiv  Detail & Related papers  (2024-05-07T23:32:36Z)
• Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts [61.929388479847525]
  This paper focuses on the problem of detecting and reacting to changes in the distribution of a sensorimotor controller's observables. The key idea is the design of switching policies that can take conformal quantiles as input. We show how to design such policies by using conformal quantiles to switch between base policies with different characteristics.
  arXiv  Detail & Related papers  (2023-11-02T17:59:30Z)
• PeRP: Personalized Residual Policies For Congestion Mitigation Through Co-operative Advisory Systems [12.010221998198423]
  Piecewise Constant (PC) Policies address issues by structurally modeling the likeness of human driving to reduce traffic congestion. We develop a co-operative advisory system based on PC policies with a novel driver trait conditioned Personalized Residual Policy, PeRP. We show that our approach successfully mitigates congestion while adapting to different driver behaviors, with 4 to 22% improvement in average speed over baselines.
  arXiv  Detail & Related papers  (2023-08-01T22:25:40Z)
• Robust Driving Policy Learning with Guided Meta Reinforcement Learning [49.860391298275616]
  We introduce an efficient method to train diverse driving policies for social vehicles as a single meta-policy. By randomizing the interaction-based reward functions of social vehicles, we can generate diverse objectives and efficiently train the meta-policy. We propose a training strategy to enhance the robustness of the ego vehicle's driving policy using the environment where social vehicles are controlled by the learned meta-policy.
  arXiv  Detail & Related papers  (2023-07-19T17:42:36Z)
• Conformal Off-Policy Evaluation in Markov Decision Processes [53.786439742572995]
  Reinforcement Learning aims at identifying and evaluating efficient control policies from data. Most methods for this learning task, referred to as Off-Policy Evaluation (OPE), do not come with accuracy and certainty guarantees. We present a novel OPE method based on Conformal Prediction that outputs an interval containing the true reward of the target policy with a prescribed level of certainty.
  arXiv  Detail & Related papers  (2023-04-05T16:45:11Z)
• Offline Reinforcement Learning with Closed-Form Policy Improvement Operators [88.54210578912554]
  Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. In this paper, we propose our closed-form policy improvement operators. We empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark.
  arXiv  Detail & Related papers  (2022-11-29T06:29:26Z)
• Mutual Information Regularized Offline Reinforcement Learning [76.05299071490913]
  We propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset. We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance.
  arXiv  Detail & Related papers  (2022-10-14T03:22:43Z)
• An Online Data-Driven Emergency-Response Method for Autonomous Agents in Unforeseen Situations [4.339510167603376]
  This paper presents an online, data-driven, emergency-response method. It aims to provide autonomous agents the ability to react to unexpected situations. We demonstrate the potential of this approach in a simulated 3D car driving scenario.
  arXiv  Detail & Related papers  (2021-12-17T18:31:37Z)
• Carl-Lead: Lidar-based End-to-End Autonomous Driving with Contrastive Deep Reinforcement Learning [10.040113551761792]
  We use deep reinforcement learning (DRL) to train lidar-based end-to-end driving policies. In this work, we use DRL to train lidar-based end-to-end driving policies that naturally consider imperfect partial observations. Our method achieves higher success rates than the state-of-the-art (SOTA) lidar-based end-to-end driving network.
  arXiv  Detail & Related papers  (2021-09-17T11:24:10Z)
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving [41.54021613421446]
  In near-accident scenarios, even a minor change in the vehicle's actions may result in drastically different consequences. We propose a hierarchical reinforcement and imitation learning (H-ReIL) approach that consists of low-level policies learned by IL for discrete driving modes, and a high-level policy learned by RL that switches between different driving modes.
  arXiv  Detail & Related papers  (2020-07-01T01:41:45Z)
• Counterfactual Policy Evaluation for Decision-Making in Autonomous Driving [3.1410342959104725]
  Learning-based approaches, such as reinforcement and imitation learning, are gaining popularity in decision-making for autonomous driving. In this work, a counterfactual policy evaluation is introduced that makes use of counterfactual worlds. We show that the proposed approach significantly decreases the collision-rate whilst maintaining a high success-rate.
  arXiv  Detail & Related papers  (2020-03-20T10:02:30Z)
• Efficient Deep Reinforcement Learning via Adaptive Policy Transfer [50.51637231309424]
  Policy Transfer Framework (PTF) is proposed to accelerate Reinforcement Learning (RL) Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods.
  arXiv  Detail & Related papers  (2020-02-19T07:30:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.