Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering

• URL: http://arxiv.org/abs/2606.29201v1
• Date: Sun, 28 Jun 2026 05:01:27 GMT
• Title: Behavior Uncloning: Distilling Mode Redirection into Policy Weights without Inference-Time Steering
• Abstract summary: We propose MoRE, which redirects policy rollouts toward desired behavior modes through a short "uncloning" step.<n>MoRE improves the average deployment success rate (SR) by 44 percentage points over the original mixed-mode policy.
• Score: 28.889110668832984
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Behavior-cloned policies often learn multiple behavior modes from demonstration datasets, including modes that are unsafe or otherwise undesired at deployment. For example, a policy trained on diverse handover demonstrations may learn to pass a knife blade-first. Standard remedies such as data curation and inference-time steering either require access to the original demonstrations for full retraining or add substantial inference-time overhead. To address this gap, we propose MoRE(Mode Redirection), which redirects policy rollouts toward desired behavior modes through a short "uncloning" step. Specifically, MoRE distills the redirection signal from a temporary mode classifier into the policy weights to steer behavior. A retain loss balances this edit by preserving desired-mode competence, allowing the standalone policy to suppress unwanted modes with zero inference-time overhead. Across eight simulated and real-world tasks, MoRE improves the average deployment success rate (SR) by 44 percentage points over the original mixed-mode policy. Among all compared adaptation and steering baselines, MoRE achieves the strongest SR and approaches the filtered-data retraining reference, while preserving task competence and inference speed. MoRE also generalizes across robot policy backbones, including Diffusion Policy and the Pi0.5 VLA, diverse task categories, and real-world deployments.

Related papers

• Proxy Policy Steering [18.880191675273796]
  Generalist robot policies carry broad manipulation priors from large-scale data, but specializing them to a new task remains the deployment bottleneck.<n>We introduce Proxy Policy Steering (PPS), an inference-time adaptation method that resolves this challenge.<n>PPS lifts the state-of-the-art pi 0.5 base policy by 53% absolute success rate on average, with zero-to-one gains on tasks the base never solves.
  arXiv  Detail & Related papers  (2026-09-08T17:59:03Z)
• LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL [64.4276583943816]
  Reinforcement learning for non-verifiable instruction relies on judges with prompt-specific rubrics as reward signals.<n>While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static.<n>This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal.
  arXiv  Detail & Related papers  (2026-07-05T17:05:13Z)
• Improving Robotic Generalist Policies via Flow Reversal Steering [92.97477771370428]
  Generalist policies can learn a wide range of skills from diverse robot datasets.<n>In order to solve or improve on challenging new tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior.<n>We propose Flow Reversal Steering (FRS): a method that takes suboptimal but reasonable'' actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes.
  arXiv  Detail & Related papers  (2026-06-11T17:59:45Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulation [20.162114513881118]
  offline reinforcement learning (RL) aims to learn the optimal policy from a fixed dataset generated by behavior policies without additional environment interactions.<n>We propose ReFORM, an offline RL method based on flow policies that enforces the less restrictive support constraint by construction.
  arXiv  Detail & Related papers  (2026-02-04T21:03:11Z)
• Learning Policy Representations for Steerable Behavior Synthesis [80.4542176039074]
  Given a Markov decision process (MDP), we seek to learn representations for a range of policies to facilitate behavior steering at test time.<n>We show that these representations can be approximated uniformly for a range of policies using a set-based architecture.<n>We use variational generative approach to induce a smooth latent space, and further shape it with contrastive learning so that latent distances align with differences in value functions.
  arXiv  Detail & Related papers  (2026-01-29T21:52:06Z)
• Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning [9.38848713730931]
  offline reinforcement learning seeks to learn optimal policies from static datasets without interacting with the environment.<n>Existing methods often assume unimodal behaviour policies, leading to suboptimal performance when this assumption is violated.<n>We propose weighted imitation Learning on One Mode (LOM), a novel approach that focuses on learning from a single, promising mode of the behaviour policy.
  arXiv  Detail & Related papers  (2024-12-04T11:57:36Z)
• Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning [13.163511229897667]
  In offline reinforcement learning, it is necessary to manage out-of-distribution actions to prevent overestimation of value functions.<n>We propose Diffusion Actor-Critic (DAC) that formulates the Kullback-Leibler (KL) constraint policy iteration as a diffusion noise regression problem.<n>Our approach is evaluated on D4RL benchmarks and outperforms the state-of-the-art in nearly all environments.
  arXiv  Detail & Related papers  (2024-05-31T00:41:04Z)
• Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints [82.43359506154117]
  We show that typical offline reinforcement learning methods fail to learn from data with non-uniform variability. Our method is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.
  arXiv  Detail & Related papers  (2022-11-02T11:36:06Z)
• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning [70.20191211010847]
  Offline reinforcement learning (RL) aims to learn an optimal policy using a previously collected static dataset. We introduce Diffusion Q-learning (Diffusion-QL) that utilizes a conditional diffusion model to represent the policy. We show that our method can achieve state-of-the-art performance on the majority of the D4RL benchmark tasks.
  arXiv  Detail & Related papers  (2022-08-12T09:54:11Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.