TakeAD: Preference-based Post-optimization for End-to-end Autonomous Driving with Expert Takeover Data

• URL: http://arxiv.org/abs/2512.17370v2
• Date: Mon, 22 Dec 2025 03:20:46 GMT
• Title: TakeAD: Preference-based Post-optimization for End-to-end Autonomous Driving with Expert Takeover Data
• Authors: Deqing Liu, Yinfeng Gao, Deheng Qian, Qichao Zhang, Xiaoqing Ye, Junyu Han, Yupeng Zheng, Xueyi Liu, Zhongpu Xia, Dawei Ding, Yifeng Pan, Dongbin Zhao,
• Abstract summary: Existing end-to-end autonomous driving methods typically rely on imitation learning (IL)<n>This misalignment often triggers driver-initiated takeovers and system disengagements during closed-loop execution.<n>We propose TakeAD, a preference-based post-optimization framework that fine-tunes the pre-trained IL policy with this disengagement data.
• Score: 40.3157492247442
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing end-to-end autonomous driving methods typically rely on imitation learning (IL) but face a key challenge: the misalignment between open-loop training and closed-loop deployment. This misalignment often triggers driver-initiated takeovers and system disengagements during closed-loop execution. How to leverage those expert takeover data from disengagement scenarios and effectively expand the IL policy's capability presents a valuable yet unexplored challenge. In this paper, we propose TakeAD, a novel preference-based post-optimization framework that fine-tunes the pre-trained IL policy with this disengagement data to enhance the closed-loop driving performance. First, we design an efficient expert takeover data collection pipeline inspired by human takeover mechanisms in real-world autonomous driving systems. Then, this post optimization framework integrates iterative Dataset Aggregation (DAgger) for imitation learning with Direct Preference Optimization (DPO) for preference alignment. The DAgger stage equips the policy with fundamental capabilities to handle disengagement states through direct imitation of expert interventions. Subsequently, the DPO stage refines the policy's behavior to better align with expert preferences in disengagement scenarios. Through multiple iterations, the policy progressively learns recovery strategies for disengagement states, thereby mitigating the open-loop gap. Experiments on the closed-loop Bench2Drive benchmark demonstrate our method's effectiveness compared with pure IL methods, with comprehensive ablations confirming the contribution of each component.

Related papers

• IPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learning [13.655904209137006]
  We propose textbfImaginary Planning Distillation (IPD), a novel framework that seamlessly incorporates offline planning into data generation, supervised training, and online inference.<n>Our framework first learns a world model equipped with uncertainty measures and a quasi-optimal value function from the offline data.<n>By replacing the conventional, manually-tuned return-to-go with the learned quasi-optimal value function, IPD improves both decision-making stability and performance during inference.
  arXiv  Detail & Related papers  (2026-03-04T17:05:39Z)
• RoaD: Rollouts as Demonstrations for Closed-Loop Supervised Fine-Tuning of Autonomous Driving Policies [30.632104005565832]
  Rollouts as Demonstrations (RoaD) is a method to mitigate covariate shift when training autonomous driving policies in closed loop.<n>During rollout generation, RoaD incorporates expert guidance to bias trajectories toward high-quality behavior, producing informative yet realistic demonstrations for fine-tuning.<n>We demonstrate the effectiveness of RoaD on WOSAC, a large-scale traffic simulation benchmark, where it performs similar or better than the prior CL-SFT method.
  arXiv  Detail & Related papers  (2025-12-01T18:52:03Z)
• Model-Based Policy Adaptation for Closed-Loop End-to-End Autonomous Driving [54.46325690390831]
  We propose Model-based Policy Adaptation (MPA), a general framework that enhances the robustness and safety of pretrained E2E driving agents during deployment.<n>MPA first generates diverse counterfactual trajectories using a geometry-consistent simulation engine.<n>MPA trains a diffusion-based policy adapter to refine the base policy's predictions and a multi-step Q value model to evaluate long-term outcomes.
  arXiv  Detail & Related papers  (2025-11-26T17:01:41Z)
• Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only [70.43369087819332]
  Supervised fine-tuning (SFT) has emerged as a crucial method for aligning large language models with human-annotated demonstrations.<n>We propose Self-Rewarding PPO, a novel fine-tuning method that leverages on-policy techniques to enhance generalization performance.
  arXiv  Detail & Related papers  (2025-10-24T02:02:13Z)
• Steerable Adversarial Scenario Generation through Test-Time Preference Alignment [58.37104890690234]
  Adversarial scenario generation is a cost-effective approach for safety assessment of autonomous driving systems.<n>We introduce a new framework named textbfSteerable textbfAdversarial scenario textbfGEnerator (SAGE)<n>SAGE enables fine-grained test-time control over the trade-off between adversariality and realism without any retraining.
  arXiv  Detail & Related papers  (2025-09-24T13:27:35Z)
• DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving [31.336758241051374]
  DriveDPO is a Safety Direct Preference Optimization Policy Learning framework.<n>We distill a unified policy distribution from human imitation similarity and rule-based safety scores for direct policy optimization.<n>Experiments on the NAVSIM benchmark demonstrate that DriveDPO achieves a new state-of-the-art PDMS of 90.0.
  arXiv  Detail & Related papers  (2025-09-22T16:01:11Z)
• SGPO: Self-Generated Preference Optimization based on Self-Improver [6.528083376369728]
  Large language models (LLMs) require alignment to human preferences for practical and reliable deployment.<n>We propose Self-Generated Preference Optimization based on Self-Improver (SGPO)<n>The improver refines responses from a policy model to self-generate preference data for direct preference optimization (DPO) of the policy model.<n> Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods.
  arXiv  Detail & Related papers  (2025-07-27T08:55:40Z)
• Forward KL Regularized Preference Optimization for Aligning Diffusion Policies [8.958830452149789]
  A central problem for learning diffusion policies is to align the policy output with human intents in various tasks.<n>We propose a novel framework, Forward KL regularized Preference optimization, to align the diffusion policy with preferences directly.<n>The results show our method exhibits superior alignment with preferences and outperforms previous state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-09-09T13:56:03Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.