Actor-Critic Pretraining for Proximal Policy Optimization

• URL: http://arxiv.org/abs/2602.23804v1
• Date: Fri, 27 Feb 2026 08:43:56 GMT
• Title: Actor-Critic Pretraining for Proximal Policy Optimization
• Abstract summary: This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO)<n>The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy.<n> Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.
• Score: 2.7575165233051693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.

Related papers

• Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition [16.480150894507908]
  We introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA)<n> Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance.<n>Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place.
  arXiv  Detail & Related papers  (2026-08-10T15:54:25Z)
• Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation [40.17737666526493]
  We propose PACT, a Preference-calibrated Actor-Critic Training framework.<n>We first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction.<n>We then build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment.
  arXiv  Detail & Related papers  (2026-06-02T17:38:25Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control [18.618486126491334]
  We find that off-policy Soft Actor-Critic (SAC) reliably supports large-scale pretraining of humanoid locomotion policies.<n>SAC-pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods.<n>Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning.
  arXiv  Detail & Related papers  (2026-01-29T07:43:24Z)
• Pretraining in Actor-Critic Reinforcement Learning for Robot Motion Control [6.288719574558261]
  This work aims to define a paradigm for pretraining neural network models.<n>We use a task-agnostic exploration-based data collection algorithm to gather diverse, dynamic transition data.<n>The pretrained weights are loaded into both the actor and critic networks to warm-start the policy optimization of actual tasks.
  arXiv  Detail & Related papers  (2025-10-14T10:25:40Z)
• Learning to Reason as Action Abstractions with Scalable Mid-Training RL [55.24192942739207]
  An effective mid-training phase should identify a compact set of useful actions and enable fast selection.<n>We propose Reasoning as Action Abstractions (RA3), a scalable mid-training algorithm.
  arXiv  Detail & Related papers  (2025-09-30T05:34:20Z)
• RLP: Reinforcement as a Pretraining Objective [103.45068938532923]
  We present an information-driven reinforcement pretraining objective that brings the core spirit of reinforcement learning -- exploration -- to the last phase of pretraining.<n>This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining.<n> Specifically, RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2025-09-26T17:53:54Z)
• What Matters for Batch Online Reinforcement Learning in Robotics? [65.06558240091758]
  The ability to learn from large batches of autonomously collected data for policy improvement holds the promise of enabling truly scalable robot learning.<n>Previous works have applied imitation learning and filtered imitation learning methods to the batch online RL problem.<n>We analyze how these axes affect performance and scaling with the amount of autonomous data.
  arXiv  Detail & Related papers  (2025-05-12T21:24:22Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• PAC-Bayesian Soft Actor-Critic Learning [9.752336113724928]
  Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm.
  arXiv  Detail & Related papers  (2023-01-30T10:44:15Z)
• Actor-Director-Critic: A Novel Deep Reinforcement Learning Framework [2.6477113498726244]
  We propose actor-director-critic, a new framework for deep reinforcement learning. For the two critic networks used, we design two target critic networks for each critic network instead of one. In order to verify the performance of the actor-director-critic framework and the improved double estimator method, we applied them to the TD3 algorithm.
  arXiv  Detail & Related papers  (2023-01-10T10:21:32Z)
• Actor Prioritized Experience Replay [0.0]
  Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. We introduce a novel experience replay sampling framework for actor-critic methods, which also regards issues with stability and recent findings behind the poor empirical performance of PER. An extensive set of experiments verifies our theoretical claims and demonstrates that the introduced method significantly outperforms the competing approaches.
  arXiv  Detail & Related papers  (2022-09-01T15:27:46Z)
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy [122.01837436087516]
  We study the global convergence and global optimality of actor-critic, one of the most popular families of reinforcement learning algorithms. We establish the rate of convergence and global optimality of single-timescale actor-critic with linear function approximation for the first time.
  arXiv  Detail & Related papers  (2020-08-02T14:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.