Reinforcement learning with distance-based incentive/penalty (DIP) updates for highly constrained industrial control systems

• URL: http://arxiv.org/abs/2011.10897v2
• Date: Wed, 19 May 2021 08:16:29 GMT
• Title: Reinforcement learning with distance-based incentive/penalty (DIP) updates for highly constrained industrial control systems
• Abstract summary: We devise a novel RL algorithm that enables an agent to handle a highly constrained action space. We apply our algorithm to an industrial control problem, microgrid system operation, and the experimental results demonstrate its superiority.
• Score: 3.3754780158324564
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Typical reinforcement learning (RL) methods show limited applicability for real-world industrial control problems because industrial systems involve various constraints and simultaneously require continuous and discrete control. To overcome these challenges, we devise a novel RL algorithm that enables an agent to handle a highly constrained action space. This algorithm has two main features. First, we devise two distance-based Q-value update schemes, incentive update and penalty update, in a distance-based incentive/penalty update technique to enable the agent to decide discrete and continuous actions in the feasible region and to update the value of these types of actions. Second, we propose a method for defining the penalty cost as a shadow price-weighted penalty. This approach affords two advantages compared to previous methods to efficiently induce the agent to not select an infeasible action. We apply our algorithm to an industrial control problem, microgrid system operation, and the experimental results demonstrate its superiority.

Related papers

• A Self-Triggered Agentic Push Recommendation System [77.13502692161426]
  Push notification is a critical recommendation scenario on large-scale platforms.<n>We propose a proactive, Self-Triggered End-to-end Agentic Push Recommendation System.<n>STEPS is already fully deployed at Douyin with over 1 billion users.
  arXiv  Detail & Related papers  (2026-08-03T09:20:53Z)
• CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning [63.5739698095413]
  Multi-task offline safe reinforcement learning (RL) promises to learn a shared optimal safe policy from offline data across multiple tasks.<n>However, the triple challenges of multi-tasking, safety constraints, and out-of-distribution actions pose a significant hurdle for existing methods.<n>We propose a Conditional Diffusion model with Contextual Prompts (CDCP) to address these challenges.
  arXiv  Detail & Related papers  (2026-07-04T14:57:35Z)
• Novel Multi-Agent Action Masked Deep Reinforcement Learning for General Industrial Assembly Lines Balancing Problems [1.8434042562191815]
  This paper introduces a novel mathematical model of a generic industrial assembly line formulated as a Markov Decision Process (MDP)<n>The proposed model is employed to create a virtual environment for training Deep Reinforcement Learning (DRL) agents to optimize task and resource scheduling.
  arXiv  Detail & Related papers  (2025-07-22T14:34:36Z)
• Efficient Beam Selection for ISAC in Cell-Free Massive MIMO via Digital Twin-Assisted Deep Reinforcement Learning [37.540612510652174]
  We derive the distribution of joint target detection probabilities across multiple receiving APs under false alarm rate constraints.<n>We then formulate the beam selection procedure as a Markov decision process (MDP)<n>To eliminate the high costs and associated risks of real-time agent-environment interactions, we propose a novel digital twin (DT)-assisted offline DRL approach.
  arXiv  Detail & Related papers  (2025-06-23T12:17:57Z)
• DARS: Dynamic Action Re-Sampling to Enhance Coding Agent Performance by Adaptive Tree Traversal [55.13854171147104]
  Large Language Models (LLMs) have revolutionized various domains, including natural language processing, data analysis, and software development. We present Dynamic Action Re-Sampling (DARS), a novel inference time compute scaling approach for coding agents. We evaluate our approach on SWE-Bench Lite benchmark, demonstrating that this scaling strategy achieves a pass@k score of 55% with Claude 3.5 Sonnet V2.
  arXiv  Detail & Related papers  (2025-03-18T14:02:59Z)
• Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs [13.443196224057658]
  Action-constrained reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation. We propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL. We show that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the state-of-the-art ACRL methods.
  arXiv  Detail & Related papers  (2025-03-17T08:41:43Z)
• Leveraging Constraint Violation Signals For Action-Constrained Reinforcement Learning [13.332006760984122]
  Action-Constrained Reinforcement Learning (ACRL) employs a projection layer after the policy network to correct the action. Recent methods were proposed to train generative models to learn a differentiable mapping between latent variables and feasible actions.
  arXiv  Detail & Related papers  (2025-02-08T12:58:26Z)
• Diffusion Predictive Control with Constraints [51.91057765703533]
  Diffusion predictive control with constraints (DPCC) An algorithm for diffusion-based control with explicit state and action constraints that can deviate from those in the training data. We show through simulations of a robot manipulator that DPCC outperforms existing methods in satisfying novel test-time constraints while maintaining performance on the learned control task.
  arXiv  Detail & Related papers  (2024-12-12T15:10:22Z)
• Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
  In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency. In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution. Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
  arXiv  Detail & Related papers  (2024-04-05T17:58:37Z)
• Variational Autoencoders for exteroceptive perception in reinforcement learning-based collision avoidance [0.0]
  Deep Reinforcement Learning (DRL) has emerged as a promising control framework. Current DRL algorithms require disproportionally large computational resources to find near-optimal policies. This paper presents a comprehensive exploration of our proposed approach in maritime control systems.
  arXiv  Detail & Related papers  (2024-03-31T09:25:28Z)
• Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications [73.58451824894568]
  The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. We propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic.
  arXiv  Detail & Related papers  (2024-02-23T23:22:06Z)
• Neural Operators for Boundary Stabilization of Stop-and-go Traffic [1.90298817989995]
  This paper introduces a novel approach to PDE boundary control design using neural operators. We present two distinct neural operator learning schemes aimed at stabilizing the traffic PDE system. It is proved that the NO-based closed-loop system is practical stable under certain approximation accuracy conditions in NO-learning.
  arXiv  Detail & Related papers  (2023-12-16T08:18:39Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning [4.67306371596399]
  Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. We show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. We propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm.
  arXiv  Detail & Related papers  (2022-11-29T10:23:26Z)
• Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models [67.78935378952146]
  GenRL is a framework for solving sequential decision-making problems. It exploits the combination of reinforcement learning and latent variable generative models. We experimentally determine the characteristics of generative models that have most influence on the performance of the final policy training.
  arXiv  Detail & Related papers  (2022-04-18T22:02:32Z)
• Softmax with Regularization: Better Value Estimation in Multi-Agent Reinforcement Learning [72.28520951105207]
  Overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning. We propose a novel regularization-based update scheme that penalizes large joint action-values deviating from a baseline. We show that our method provides a consistent performance improvement on a set of challenging StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2021-03-22T14:18:39Z)
• A Dynamic Penalty Function Approach for Constraints-Handling in Reinforcement Learning [0.0]
  This study focuses on usingReinforcement learning (RL) to solve constrained optimal control problems. While training neural networks to learn the value (or Q) function, one can run into computational issues caused by the sharp change in the function value at the constraint boundary. This difficulty during training can lead to convergence problems and ultimately lead to poor closed-loop performance.
  arXiv  Detail & Related papers  (2020-12-22T02:13:59Z)
• Constrained Markov Decision Processes via Backward Value Functions [43.649330976089004]
  We model the problem of learning with constraints as a Constrained Markov Decision Process. A key contribution of our approach is to translate cumulative cost constraints into state-based constraints. We provide theoretical guarantees under which the agent converges while ensuring safety over the course of training.
  arXiv  Detail & Related papers  (2020-08-26T20:56:16Z)
• Combining Deep Learning and Optimization for Security-Constrained Optimal Power Flow [94.24763814458686]
  Security-constrained optimal power flow (SCOPF) is fundamental in power systems. Modeling of APR within the SCOPF problem results in complex large-scale mixed-integer programs. This paper proposes a novel approach that combines deep learning and robust optimization techniques.
  arXiv  Detail & Related papers  (2020-07-14T12:38:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.