Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning

• URL: http://arxiv.org/abs/2512.16911v1
• Date: Thu, 18 Dec 2025 18:59:17 GMT
• Title: Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• Abstract summary: We first show theoretically that standard behavioral cloning (BC) can fail to ensure coverage over the demonstrator's actions.<n>We then show that if, instead of exactly fitting the observed demonstrations, we train a policy to model the posterior distribution of the demonstrator's behavior.<n>This policy ensures coverage over the demonstrator's actions, enabling more effective finetuning.
• Score: 87.81738284453013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard practice across domains from robotics to language is to first pretrain a policy on a large-scale demonstration dataset, and then finetune this policy, typically with reinforcement learning (RL), in order to improve performance on deployment domains. This finetuning step has proved critical in achieving human or super-human performance, yet while much attention has been given to developing more effective finetuning algorithms, little attention has been given to ensuring the pretrained policy is an effective initialization for RL finetuning. In this work we seek to understand how the pretrained policy affects finetuning performance, and how to pretrain policies in order to ensure they are effective initializations for finetuning. We first show theoretically that standard behavioral cloning (BC) -- which trains a policy to directly match the actions played by the demonstrator -- can fail to ensure coverage over the demonstrator's actions, a minimal condition necessary for effective RL finetuning. We then show that if, instead of exactly fitting the observed demonstrations, we train a policy to model the posterior distribution of the demonstrator's behavior given the demonstration dataset, we do obtain a policy that ensures coverage over the demonstrator's actions, enabling more effective finetuning. Furthermore, this policy -- which we refer to as the posterior behavioral cloning (PostBC) policy -- achieves this while ensuring pretrained performance is no worse than that of the BC policy. We then show that PostBC is practically implementable with modern generative models in robotic control domains -- relying only on standard supervised learning -- and leads to significantly improved RL finetuning performance on both realistic robotic control benchmarks and real-world robotic manipulation tasks, as compared to standard behavioral cloning.

Related papers

• Expert Behavior Prior Reinforcement Learning [10.983482150597913]
  Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL)<n>Most existing BPRL methods rely on static offline datasets, which often suffer from low data diversity and suboptimal trajectory quality.<n>We propose an Expert Behavior Prior (EBP) algorithm that learns to generate expert policy priors directly from the online replay buffer.<n>EBP significantly outperforms state-of-the-art online RL algorithms, achieving higher sample efficiency and more stable convergence.
  arXiv  Detail & Related papers  (2026-07-23T13:26:57Z)
• Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards [24.576576709809036]
  Reinforcement learning can be used to finetune policies further using additional experience.<n>In inverse reinforcement learning, a dense reward function is learned from expert demonstrations.<n>We show that our IRL method maintains or improves the performance of pi-0.5 on all six sparse manipulation tasks and achieves a $geq 90%$ success rate on five out of six complex manipulation tasks.
  arXiv  Detail & Related papers  (2026-06-01T12:49:14Z)
• TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning [9.021212644908767]
  Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral cloning (BC)<n>We present a unified framework that enables the exploration necessary to enable efficient robot policy finetuning.
  arXiv  Detail & Related papers  (2026-05-12T15:07:04Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• Update-Free On-Policy Steering via Verifiers [47.56134859358476]
  Behavior Cloning (BC) has become one of the most prevalent methods for enabling robots to mimic human demonstrations.<n>We propose UF-OPS, an Update-Free On-Policy Steering method that enables the robot to predict the success likelihood of its actions.<n>We present results from both simulation and real-world data and achieve an average 49% improvement in success rate over the base policy across 5 real tasks.
  arXiv  Detail & Related papers  (2026-03-10T23:55:52Z)
• EXPO: Stable Reinforcement Learning with Expressive Policies [74.30151915786233]
  We propose a sample-efficient online reinforcement learning algorithm to maximize value with two parameterized policies.<n>Our approach yields up to 2-3x improvement in sample efficiency on average over prior methods.
  arXiv  Detail & Related papers  (2025-07-10T17:57:46Z)
• Steering Your Diffusion Policy with Latent Space Reinforcement Learning [46.598122553180005]
  Behavioral cloning (BC)-learned policies typically require collecting additional human demonstrations to further improve their behavior.<n> reinforcement learning (RL) holds the promise of enabling autonomous online policy improvement, but often falls short of achieving this due to the large number of samples it typically requires.<n>We show that DSRL is highly sample efficient, requires only black-box access to the BC policy, and enables effective real-world autonomous policy improvement.
  arXiv  Detail & Related papers  (2025-06-18T18:35:57Z)
• Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models [71.34520793462069]
  Unsupervised reinforcement learning (RL) aims at pre-training agents that can solve a wide range of downstream tasks in complex environments.<n>We introduce a novel algorithm regularizing unsupervised RL towards imitating trajectories from unlabeled behavior datasets.<n>We demonstrate the effectiveness of this new approach in a challenging humanoid control problem.
  arXiv  Detail & Related papers  (2025-04-15T10:41:11Z)
• FDPP: Fine-tune Diffusion Policy with Human Preference [57.44575105114056]
  Fine-tuning Diffusion Policy with Human Preference learns a reward function through preference-based learning.<n>This reward is then used to fine-tune the pre-trained policy with reinforcement learning.<n>Experiments demonstrate that FDPP effectively customizes policy behavior without compromising performance.
  arXiv  Detail & Related papers  (2025-01-14T17:15:27Z)
• Foundation Policies with Hilbert Representations [54.44869979017766]
  We propose an unsupervised framework to pre-train generalist policies from unlabeled offline data. Our key insight is to learn a structured representation that preserves the temporal structure of the underlying environment. Our experiments show that our unsupervised policies can solve goal-conditioned and general RL tasks in a zero-shot fashion.
  arXiv  Detail & Related papers  (2024-02-23T19:09:10Z)
• Offline Reinforcement Learning with Closed-Form Policy Improvement Operators [88.54210578912554]
  Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. In this paper, we propose our closed-form policy improvement operators. We empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark.
  arXiv  Detail & Related papers  (2022-11-29T06:29:26Z)
• Offline Reinforcement Learning with Soft Behavior Regularization [0.8937096931077437]
  In this work, we derive a new policy learning objective that can be used in the offline setting. Unlike state-independent regularization used in prior approaches, this textitsoft regularization allows more freedom of policy deviation. Our experimental results show that SBAC matches or outperforms the state-of-the-art on a set of continuous control locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2021-10-14T14:29:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.