Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control

• URL: http://arxiv.org/abs/2604.03023v1
• Date: Fri, 03 Apr 2026 13:13:47 GMT
• Title: Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control
• Abstract summary: Reinforcement learning can discover high-performing strategies but often departs from desirable human behavior.<n>We propose a behavior-constrained reinforcement learning framework that improves beyond demonstrations.<n>We evaluate the approach in high-fidelity race car simulation using data from professional drivers.
• Score: 13.52467502450336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning high-performance control policies that remain consistent with expert behavior is a fundamental challenge in robotics. Reinforcement learning can discover high-performing strategies but often departs from desirable human behavior, whereas imitation learning is limited by demonstration quality and struggles to improve beyond expert data. We propose a behavior-constrained reinforcement learning framework that improves beyond demonstrations while explicitly controlling deviation from expert behavior. Because expert-consistent behavior in dynamic control is inherently trajectory-level, we introduce a receding-horizon predictive mechanism that models short-term future trajectories and provides look-ahead rewards during training. To account for the natural variability of human behavior under disturbances and changing conditions, we further condition the policy on reference trajectories, allowing it to represent a distribution of expert-consistent behaviors rather than a single deterministic target. Empirically, we evaluate the approach in high-fidelity race car simulation using data from professional drivers, a domain characterized by extreme dynamics and narrow performance margins. The learned policies achieve competitive lap times while maintaining close alignment with expert driving behavior, outperforming baseline methods in both performance and imitation quality. Beyond standard benchmarks, we conduct human-grounded evaluation in a driver-in-the-loop simulator and show that the learned policies reproduce setup-dependent driving characteristics consistent with the feedback of top-class professional race drivers. These results demonstrate that our method enables learning high-performance control policies that are both optimal and behavior-consistent, and can serve as reliable surrogates for human decision-making in complex control systems.

Related papers

• Self-Aware Active Learning Enables Continual Improvement in Autonomous Driving [12.669051115615176]
  Self-aware guided exploration (SAGE) is an active learning framework for post-training adaptation in autonomous driving.<n>Curiosity adaptively calibrates the intervention threshold for fear, allowing the agent to regulate risk in a context-dependent manner.<n>Fear is integrated into policy optimization and evaluation as a safety-oriented constraint to reduce performance regressions.
  arXiv  Detail & Related papers  (2026-08-30T13:01:32Z)
• Expert Knowledge-driven Reinforcement Learning for Autonomous Racing via Trajectory Guidance and Dynamics Constraints [39.69523326789163]
  This paper proposes a guidance trajectory and dynamics constraints Reinforcement Learning (TraD-RL) method for autonomous racing.<n>The proposed method is evaluated in a high-fidelity simulation environment modeled after the Tempelhof Airport Street Circuit.<n> Experimental results demonstrate that TraD-RL effectively improves both lap speed and driving stability of the autonomous racing vehicle.
  arXiv  Detail & Related papers  (2026-03-06T02:51:54Z)
• Post-Training and Test-Time Scaling of Generative Agent Behavior Models for Interactive Autonomous Driving [3.8612647047433217]
  Group Relative Behavior Optimization improves safety performance by over 40% while preserving behavioral realism.<n>Warm-K is a warm-started Top-K sampling strategy that balances consistency and diversity in motion selection.
  arXiv  Detail & Related papers  (2025-12-15T12:18:50Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Can Large Reasoning Models Self-Train? [51.0277533541394]
  We use majority voting as a simple self-feedback mechanism to study whether self-training can be sustained within reinforcement learning.<n>We find that this basic approach improves not only the model's reasoning performance, but also its capability of generating better quality feedback for the next RL iteration.<n>Yet our analysis also reveals a critical limitation of such a self-training paradigm - prolonged RL with self-reward leads to reward hacking, resulting in sudden and complete performance collapse.
  arXiv  Detail & Related papers  (2025-05-27T17:16:00Z)
• Discrete Contrastive Learning for Diffusion Policies in Autonomous Driving [18.624545462468642]
  We propose a novel approach that leverages contrastive learning to extract a dictionary of driving styles from pre-existing human driving data.<n>Our empirical evaluation confirms that the behaviors generated by our approach are both safer and more human-like than those of the machine-learning-based baseline methods.
  arXiv  Detail & Related papers  (2025-03-07T08:26:04Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Imitation Learning by State-Only Distribution Matching [2.580765958706854]
  Imitation Learning from observation describes policy learning in a similar way to human learning. We propose a non-adversarial learning-from-observations approach, together with an interpretable convergence and performance metric.
  arXiv  Detail & Related papers  (2022-02-09T08:38:50Z)
• Closing the Closed-Loop Distribution Shift in Safe Imitation Learning [80.05727171757454]
  We treat safe optimization-based control strategies as experts in an imitation learning problem. We train a learned policy that can be cheaply evaluated at run-time and that provably satisfies the same safety guarantees as the expert.
  arXiv  Detail & Related papers  (2021-02-18T05:11:41Z)
• Behaviorally Diverse Traffic Simulation via Reinforcement Learning [16.99423598448411]
  This paper introduces an easily-tunable policy generation algorithm for autonomous driving agents. The proposed algorithm balances diversity and driving skills by leveraging the representation and exploration abilities of deep reinforcement learning. We experimentally show the effectiveness of our methods on several challenging intersection scenes.
  arXiv  Detail & Related papers  (2020-11-11T12:49:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.