Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces

• URL: http://arxiv.org/abs/2306.01896v3
• Date: Sun, 26 May 2024 23:47:48 GMT
• Title: Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces
• Abstract summary: In many reinforcement learning (RL) applications, we want policies that reach desired states and keep the controlled system within an acceptable region. In this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies. We introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space.
• Score: 13.544995860887298
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. This latter objective is called stability and is especially important when the state space is unbounded, such that the states can be arbitrarily far from each other and the agent can drift far away from the desired states. For example, in stochastic queuing networks, where queues of waiting jobs can grow without bound, the desired state is all-zero queue lengths. Here, a stable policy ensures queue lengths are finite while an optimal policy minimizes queue lengths. Since an optimal policy is also stable, one would expect that RL algorithms would implicitly give us stable policies. However, in this work, we find that deep RL algorithms that directly minimize the distance to the desired state during online training often result in unstable policies, i.e., policies that drift far away from the desired state. We attribute this instability to poor credit-assignment for destabilizing actions. We then introduce an approach based on two ideas: 1) a Lyapunov-based cost-shaping technique and 2) state transformations to the unbounded state space. We conduct an empirical study on various queueing networks and traffic signal control problems and find that our approach performs competitively against strong baselines with knowledge of the transition dynamics. Our code is available here: https://github.com/Badger-RL/STOP.

Related papers

• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximation [2.5291809836356998]
  We introduce a proximal policy to decouple the off-policy corrections from the controlling policy updates.<n>The policy requires an extra forward pass through the network at each training step, creating a computational bottleneck.<n>We observe that since the proximal policy only serves as a trust region anchor between the behavior and target policies, we can approximate it through simple without explicit computation.
  arXiv  Detail & Related papers  (2025-12-06T19:37:39Z)
• Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning [13.831084892489754]
  In natural situations, it is desirable to learn a policy that induces a dispersed marginal state distribution over rewarding states.<n>We propose a novel algorithm that learns a high-return policy mixture with marginal state distribution dispersed over the set of goal states.
  arXiv  Detail & Related papers  (2025-10-29T09:23:21Z)
• Fat-to-Thin Policy Optimization: Offline RL with Sparse Policies [5.5938591697033555]
  Sparse continuous policies are distributions that choose some actions at random yet keep strictly zero probability for the other actions. In this paper, we propose the first offline policy optimization algorithm that tackles this challenge: Fat-to-Thin Policy Optimization (FtTPO) We instantiate FtTPO with the general $q$-Gaussian family that encompasses both heavy-tailed and sparse policies.
  arXiv  Detail & Related papers  (2025-01-24T10:11:48Z)
• Decision-Point Guided Safe Policy Improvement [22.885394395400592]
  Decision Points RL (DPRL) is an algorithm that restricts the set of state-action pairs (or regions for continuous states) considered for improvement. DPRL ensures high-confidence improvement in densely visited states while still utilizing data from sparsely visited states.
  arXiv  Detail & Related papers  (2024-10-12T04:05:56Z)
• Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning [71.02384943570372]
  Family Offline-to-Online RL (FamO2O) is a framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark.
  arXiv  Detail & Related papers  (2023-10-27T08:30:54Z)
• Policy Regularization with Dataset Constraint for Offline Reinforcement Learning [27.868687398300658]
  We consider the problem of learning the best possible policy from a fixed dataset, known as offline Reinforcement Learning (RL) In this paper, we find that regularizing the policy towards the nearest state-action pair can be more effective and thus propose Policy Regularization with dataset Constraint (PRDC) PRDC can guide the policy with proper behaviors from the dataset, allowing it to choose actions that do not appear in the dataset along with the given state.
  arXiv  Detail & Related papers  (2023-06-11T03:02:10Z)
• Feasible Policy Iteration [28.29623882912745]
  We propose an indirect safe RL framework called feasible policy iteration. It guarantees that the feasible region monotonically expands and converges to the maximum one. Experiments show that our algorithm learns strictly safe and near-optimal policies with accurate feasible regions.
  arXiv  Detail & Related papers  (2023-04-18T09:18:37Z)
• Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints [82.43359506154117]
  We show that typical offline reinforcement learning methods fail to learn from data with non-uniform variability. Our method is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.
  arXiv  Detail & Related papers  (2022-11-02T11:36:06Z)
• A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning [61.406020873047794]
  A major hurdle to real-world application arises from the development of algorithms in an episodic setting. We propose a new method, MEDAL, that trains the backward policy to match the state distribution in the provided demonstrations. Our experiments show that MEDAL matches or outperforms prior methods on three sparse-reward continuous control tasks.
  arXiv  Detail & Related papers  (2022-05-11T00:06:29Z)
• Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization [65.09725599705493]
  We propose to chain multiple policies without excessively large initial state distributions. We evaluate our approach on two complex long-horizon manipulation tasks of furniture assembly. Our results have shown that our method establishes the first model-free reinforcement learning algorithm to solve these tasks.
  arXiv  Detail & Related papers  (2021-11-15T18:59:03Z)
• Stable Reinforcement Learning with Unbounded State Space [27.053432445897016]
  We consider the problem of reinforcement learning with unbounded state space motivated by the classical problem of scheduling in a queueing network. Traditional policies as well as error metric that are designed for finite, bounded or compact state space, require infinite samples for providing meaningful performance guarantee. We propose stability as the notion of "goodness": the state dynamics under the policy should remain in a bounded region with high probability.
  arXiv  Detail & Related papers  (2020-06-08T05:00:25Z)
• Deep Reinforcement Learning with Robust and Smooth Policy [90.78795857181727]
  We propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- textbfSmooth textbfRegularized textbfReinforcement textbfLearning ($textbfSR2textbfL$), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy.
  arXiv  Detail & Related papers  (2020-03-21T00:10:29Z)
• BRPO: Batch Residual Policy Optimization [79.53696635382592]
  In batch reinforcement learning, one often constrains a learned policy to be close to the behavior (data-generating) policy. We propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method, BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance.
  arXiv  Detail & Related papers  (2020-02-08T01:59:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.