Related papers: MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins

MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins

URL: http://arxiv.org/abs/2606.26392v1
Date: Wed, 24 Jun 2026 21:23:22 GMT
Title: MPC-Injection: Biasing Off-Policy Locomotion RL Toward Controller-Induced Behavior Basins
Abstract summary: We present MPC-Injection, a low-overhead method that drives the policy into the controller's behavior basin.<n>Unlike reward shaping, MPC-Injection does not require redesigning the task reward.<n>We show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward.
Score: 7.729687505744033
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Reinforcement learning (RL) for locomotion frequently converges to locally optimal but undeployable behaviors, such as vibrating limbs or scooting on the torso, that maximize return without producing a usable gait. We present MPC-Injection, a low-overhead method that steers RL toward a designer-preferred gait by inserting transitions into the replay buffer from a model predictive controller solving the same Markov decision process. Unlike reward shaping, MPC-Injection does not require redesigning the task reward, and unlike adversarial imitation learning, it adds no discriminator, no kinematic retargeting, and no auxiliary objective. Instead, the controller's preferred behavior is transferred to the policy purely through the replay state distribution. On a 2D walker in simulation and with sim-to-real evaluation on a Go2 quadruped, we show that MPC-Injection drives the policy into the controller's behavior basin using a one to two-term task reward, producing gaits qualitatively comparable to those of reward shaping with twenty-one tuned terms and of adversarial motion priors without their discriminator and retargeting overhead. We further analyze how the injected transitions bias actor-critic updates toward controller-visited states, allowing the policy to learn behaviors that pure RL may fail to reach under simple reward functions.

Related papers

RGB: RL Guided Whole-Body MPPI for Humanoid Control [2.854451361373021]
We propose an RL guided whole-body model predictive path integral (MPPI) framework that acts as an add-on feedback controller.<n>The proposed method improves task-level precision over a pure RL baseline under the same command interface.
arXiv Detail & Related papers (2026-06-23T19:54:02Z)
Probabilistic Recurrent Intention Switching Model [8.25071593831945]
Inverse reinforcement learning (IRL) recovers reward functions from observed behavior, yet traditional methods assume a single stationary reward that cannot capture goal switching within an episode.<n>Recent multi-intention IRL methods address this by segmenting trajectories, but model intention transitions as either a memoryless Markov chain or via manual state augmentation with a fixed history window.<n>We propose the Probabilistic Recurrent Intention Switching Model (PRISM), which replaces both mechanisms with a lightweight recurrent network that maps observation history to a per-step intention distribution.
arXiv Detail & Related papers (2026-05-26T13:19:00Z)
Insect-inspired modular architectures as inductive biases for reinforcement learning [0.0]
We study an RL policy architecture that decomposes control into interacting modules for sensory encoding, heading representation, sparse associative memory, recurrent command generation, and local motor control.<n>The model is evaluated on a two-dimensional navigation task that require simultaneous food seeking, obstacle avoidance, and predator escape.
arXiv Detail & Related papers (2026-04-23T21:26:31Z)
Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
arXiv Detail & Related papers (2026-03-07T01:49:59Z)
Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
arXiv Detail & Related papers (2026-01-26T21:38:20Z)
SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
arXiv Detail & Related papers (2025-10-20T19:53:02Z)
Double Check My Desired Return: Transformer with Target Alignment for Offline Reinforcement Learning [64.6334337560557]
Reinforcement learning via supervised learning (RvS) frames offline RL as a sequence modeling task.<n>Decision Transformer (DT) struggles to reliably align the actual achieved returns with specified target returns.<n>We propose Doctor, a novel approach that Double Checks the Transformer with target alignment for Offline RL.
arXiv Detail & Related papers (2025-08-22T14:30:53Z)
Post-Convergence Sim-to-Real Policy Transfer: A Principled Alternative to Cherry-Picking [5.027571997864706]
This paper addresses the post-convergence sim-to-real transfer problem by introducing a worst-case performance transference optimization approach.<n>Experiments demonstrate its effectiveness in transferring RL-based locomotion policies from simulation to real-world laboratory tests.
arXiv Detail & Related papers (2025-04-21T19:48:05Z)
Optimal Controller Realizations against False Data Injections in Cooperative Driving [2.2134894590368748]
We study a controller-oriented approach to mitigate the effect of a class of False-Data Injection (FDI) attacks. We show that a class of new but equivalent controllers can represent the base controller. We obtain the optimal combination of sensors that minimizes the effect of FDI attacks.
arXiv Detail & Related papers (2024-04-08T09:53:42Z)
REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
arXiv Detail & Related papers (2023-12-22T04:56:37Z)
Residual Reinforcement Learning from Demonstrations [51.56457466788513]
Residual reinforcement learning (RL) has been proposed as a way to solve challenging robotic tasks by adapting control actions from a conventional feedback controller to maximize a reward signal. We extend the residual formulation to learn from visual inputs and sparse rewards using demonstrations. Our experimental evaluation on simulated manipulation tasks on a 6-DoF UR5 arm and a 28-DoF dexterous hand demonstrates that residual RL from demonstrations is able to generalize to unseen environment conditions more flexibly than either behavioral cloning or RL fine-tuning.
arXiv Detail & Related papers (2021-06-15T11:16:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.