From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning

• URL: http://arxiv.org/abs/2603.10263v1
• Date: Tue, 10 Mar 2026 22:49:46 GMT
• Title: From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
• Abstract summary: DICE-RL is a framework that uses reinforcement learning to refine pretrained generative robot policies.<n>It turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback.<n>Experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency.
• Score: 17.846597717733058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Distribution Contractive Reinforcement Learning (DICE-RL), a framework that uses reinforcement learning (RL) as a "distribution contraction" operator to refine pretrained generative robot policies. DICE-RL turns a pretrained behavior prior into a high-performing "pro" policy by amplifying high-success behaviors from online feedback. We pretrain a diffusion- or flow-based policy for broad behavioral coverage, then finetune it with a stable, sample-efficient residual off-policy RL framework that combines selective behavior regularization with value-guided action selection. Extensive experiments and analyses show that DICE-RL reliably improves performance with strong stability and sample efficiency. It enables mastery of complex long-horizon manipulation skills directly from high-dimensional pixel inputs, both in simulation and on a real robot. Project website: https://zhanyisun.github.io/dice.rl.2026/.

Related papers

• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning [38.54413500261524]
  Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions.<n>We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills.
  arXiv  Detail & Related papers  (2026-07-16T09:57:18Z)
• Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards [24.576576709809036]
  Reinforcement learning can be used to finetune policies further using additional experience.<n>In inverse reinforcement learning, a dense reward function is learned from expert demonstrations.<n>We show that our IRL method maintains or improves the performance of pi-0.5 on all six sparse manipulation tasks and achieves a $geq 90%$ success rate on five out of six complex manipulation tasks.
  arXiv  Detail & Related papers  (2026-06-01T12:49:14Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models [73.10315509190623]
  Recent reinforcement learning techniques have yielded impressive reasoning improvements in language models.<n>It remains unclear whether post-training truly extends a model's reasoning ability beyond what it acquires during pre-training.<n>We develop a fully controlled experimental framework that isolates the causal contributions of pre-training, mid-training, and RL-based post-training.
  arXiv  Detail & Related papers  (2025-12-08T18:12:10Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Learning to Reason as Action Abstractions with Scalable Mid-Training RL [55.24192942739207]
  An effective mid-training phase should identify a compact set of useful actions and enable fast selection.<n>We propose Reasoning as Action Abstractions (RA3), a scalable mid-training algorithm.
  arXiv  Detail & Related papers  (2025-09-30T05:34:20Z)
• Residual Off-Policy RL for Finetuning Behavior Cloning Policies [41.99435186991878]
  We present a recipe that combines the benefits of behavior cloning (BC) and reinforcement learning (RL) through a residual learning framework.<n>Our method requires only sparse binary reward signals and can effectively improve manipulation policies on high-degree-of-freedom (DoF) systems.<n>In particular, we demonstrate, to the best of our knowledge, the first successful real-world RL training on a humanoid robot with dexterous hands.
  arXiv  Detail & Related papers  (2025-09-23T17:59:46Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences [23.414135977983953]
  Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. We present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences.
  arXiv  Detail & Related papers  (2024-02-27T07:03:25Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.