From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving

• URL: http://arxiv.org/abs/2508.07029v2
• Date: Wed, 27 Aug 2025 14:32:13 GMT
• Title: From Imitation to Optimization: A Comparative Study of Offline Learning for Autonomous Driving
• Authors: Antonio Guillen-Perez,
• Abstract summary: This work presents a comprehensive pipeline and a comparative study to address this limitation.<n>We first develop a series of increasingly sophisticated Behavioral Cloning (BC) baselines.<n>We then demonstrate that by applying a state-of-the-art Offline Reinforcement Learning algorithm, Conservative Q-Learning (CQL), to the same data and architecture, we can learn a significantly more robust policy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning robust driving policies from large-scale, real-world datasets is a central challenge in autonomous driving, as online data collection is often unsafe and impractical. While Behavioral Cloning (BC) offers a straightforward approach to imitation learning, policies trained with BC are notoriously brittle and suffer from compounding errors in closed-loop execution. This work presents a comprehensive pipeline and a comparative study to address this limitation. We first develop a series of increasingly sophisticated BC baselines, culminating in a Transformer-based model that operates on a structured, entity-centric state representation. While this model achieves low imitation loss, we show that it still fails in long-horizon simulations. We then demonstrate that by applying a state-of-the-art Offline Reinforcement Learning algorithm, Conservative Q-Learning (CQL), to the same data and architecture, we can learn a significantly more robust policy. Using a carefully engineered reward function, the CQL agent learns a conservative value function that enables it to recover from minor errors and avoid out-of-distribution states. In a large-scale evaluation on 1,000 unseen scenarios from the Waymo Open Motion Dataset, our final CQL agent achieves a 3.2x higher success rate and a 7.4x lower collision rate than the strongest BC baseline, proving that an offline RL approach is critical for learning robust, long-horizon driving policies from static expert data.

Related papers

• Mining the Long Tail: A Comparative Study of Data-Centric Criticality Metrics for Robust Offline Reinforcement Learning in Autonomous Motion Planning [0.0]
  We study data curation strategies to focus the learning process on information-rich samples.<n>We train seven goal-conditioned Conservative Q-Learning (CQL) agents with a state-of-the-art, attention-based architecture.<n>Data-driven curation using model uncertainty as a signal achieves the most significant safety improvements.
  arXiv  Detail & Related papers  (2025-08-25T18:37:29Z)
• Strategically Conservative Q-Learning [89.17906766703763]
  offline reinforcement learning (RL) is a compelling paradigm to extend RL's practical utility. The major difficulty in offline RL is mitigating the impact of approximation errors when encountering out-of-distribution (OOD) actions. We propose a novel framework called Strategically Conservative Q-Learning (SCQ) that distinguishes between OOD data that is easy and hard to estimate.
  arXiv  Detail & Related papers  (2024-06-06T22:09:46Z)
• Towards Robust Offline Reinforcement Learning under Diverse Data Corruption [46.16052026620402]
  We show that implicit Q-learning (IQL) demonstrates remarkable resilience to data corruption among various offline RL algorithms. We propose a more robust offline RL approach named Robust IQL (RIQL)
  arXiv  Detail & Related papers  (2023-10-19T17:54:39Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Contextual Conservative Q-Learning for Offline Reinforcement Learning [15.819356579361843]
  We propose Contextual Conservative Q-Learning(C-CQL) to learn a robustly reliable policy through the contextual information captured via an inverse dynamics model. C-CQL achieves the state-of-the-art performance in most environments of offline Mujoco suite and a noisy Mujoco setting.
  arXiv  Detail & Related papers  (2023-01-03T13:33:54Z)
• Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints [82.43359506154117]
  We show that typical offline reinforcement learning methods fail to learn from data with non-uniform variability. Our method is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.
  arXiv  Detail & Related papers  (2022-11-02T11:36:06Z)
• FIRE: A Failure-Adaptive Reinforcement Learning Framework for Edge Computing Migrations [52.85536740465277]
  FIRE is a framework that adapts to rare events by training a RL policy in an edge computing digital twin environment. We propose ImRE, an importance sampling-based Q-learning algorithm, which samples rare events proportionally to their impact on the value function. We show that FIRE reduces costs compared to vanilla RL and the greedy baseline in the event of failures.
  arXiv  Detail & Related papers  (2022-09-28T19:49:39Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• Overcoming Model Bias for Robust Offline Deep Reinforcement Learning [3.1325640909772403]
  MOOSE is an algorithm which ensures low model bias by keeping the policy within the support of the data. We compare MOOSE with state-of-the-art model-free, offline RL algorithms BRAC, BEAR and BCQ on the Industrial Benchmark and MuJoCo continuous control tasks in terms of robust performance.
  arXiv  Detail & Related papers  (2020-08-12T19:08:55Z)
• Conservative Q-Learning for Offline Reinforcement Learning [106.05582605650932]
  We show that CQL substantially outperforms existing offline RL methods, often learning policies that attain 2-5 times higher final return. We theoretically show that CQL produces a lower bound on the value of the current policy and that it can be incorporated into a policy learning procedure with theoretical improvement guarantees.
  arXiv  Detail & Related papers  (2020-06-08T17:53:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.