Temporally Layered Architecture for Efficient Continuous Control
- URL: http://arxiv.org/abs/2305.18701v2
- Date: Wed, 9 Aug 2023 02:09:15 GMT
- Title: Temporally Layered Architecture for Efficient Continuous Control
- Authors: Devdhar Patel, Terrence Sejnowski, Hava Siegelmann
- Abstract summary: We present a temporally layered architecture (TLA) for temporally adaptive control with minimal energy expenditure.
Our design draws on the energy-saving mechanism of the human brain, which executes actions at different timescales depending on the environment's demands.
- Score: 1.933681537640272
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: We present a temporally layered architecture (TLA) for temporally adaptive
control with minimal energy expenditure. The TLA layers a fast and a slow
policy together to achieve temporal abstraction that allows each layer to focus
on a different time scale. Our design draws on the energy-saving mechanism of
the human brain, which executes actions at different timescales depending on
the environment's demands. We demonstrate that beyond energy saving, TLA
provides many additional advantages, including persistent exploration, fewer
required decisions, reduced jerk, and increased action repetition. We evaluate
our method on a suite of continuous control tasks and demonstrate the
significant advantages of TLA over existing methods when measured over multiple
important metrics. We also introduce a multi-objective score to qualitatively
assess continuous control policies and demonstrate a significantly better score
for TLA. Our training algorithm uses minimal communication between the slow and
fast layers to train both policies simultaneously, making it viable for future
applications in distributed control.
Related papers
- Temporal Prototype-Aware Learning for Active Voltage Control on Power Distribution Networks [28.630650305620197]
Active Voltage Control (AVC) on the Power Distribution Networks (PDNs) aims to stabilize the voltage levels to ensure efficient and reliable operation of power systems.
We propose a novel temporal prototype-aware learning method, abbreviated as TPA, to learn time-adaptive dependencies under short-term training trajectories.
arXiv Detail & Related papers (2024-06-25T08:07:00Z) - When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL [37.58940726230092]
Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP)
We formalize an RL framework, Time-adaptive Control & Sensing (TaCoS), that tackles this challenge.
We demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart.
arXiv Detail & Related papers (2024-06-03T09:57:18Z) - Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution [51.83951489847344]
In robotics applications, smooth control signals are commonly preferred to reduce system wear and energy efficiency.
In this work, we aim to bridge this performance gap by growing discrete action spaces from coarse to fine control resolution.
Our work indicates that an adaptive control resolution in combination with value decomposition yields simple critic-only algorithms that yield surprisingly strong performance on continuous control tasks.
arXiv Detail & Related papers (2024-04-05T17:58:37Z) - Action-Quantized Offline Reinforcement Learning for Robotic Skill
Learning [68.16998247593209]
offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data.
In this paper, we propose an adaptive scheme for action quantization.
We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
arXiv Detail & Related papers (2023-10-18T06:07:10Z) - Non-Stationary Policy Learning for Multi-Timescale Multi-Agent
Reinforcement Learning [9.808555135836022]
In multi-timescale multi-agent reinforcement learning, agents interact across different timescales.
We introduce a simple framework for learning non-stationary policies for multi-timescale MARL.
The framework's ability to effectively learn multi-timescale policies is validated on a gridworld and building energy management environment.
arXiv Detail & Related papers (2023-07-17T19:25:46Z) - Long-term Wind Power Forecasting with Hierarchical Spatial-Temporal
Transformer [112.12271800369741]
Wind power is attracting increasing attention around the world due to its renewable, pollution-free, and other advantages.
Accurate wind power forecasting (WPF) can effectively reduce power fluctuations in power system operations.
Existing methods are mainly designed for short-term predictions and lack effective spatial-temporal feature augmentation.
arXiv Detail & Related papers (2023-05-30T04:03:15Z) - Temporally Layered Architecture for Adaptive, Distributed and Continuous
Control [2.1700103865910503]
We present temporally layered architecture (TLA), a biologically inspired system for temporally adaptive distributed control.
TLA layers a fast and a slow controller together to achieve temporal abstraction that allows each layer to focus on a different time-scale.
Our design is biologically inspired and draws on the architecture of the human brain which executes actions at different timescales depending on the environment's demands.
arXiv Detail & Related papers (2022-12-25T08:46:22Z) - Distributed-Training-and-Execution Multi-Agent Reinforcement Learning
for Power Control in HetNet [48.96004919910818]
We propose a multi-agent deep reinforcement learning (MADRL) based power control scheme for the HetNet.
To promote cooperation among agents, we develop a penalty-based Q learning (PQL) algorithm for MADRL systems.
In this way, an agent's policy can be learned by other agents more easily, resulting in a more efficient collaboration process.
arXiv Detail & Related papers (2022-12-15T17:01:56Z) - Accelerated Reinforcement Learning for Temporal Logic Control Objectives [10.216293366496688]
This paper addresses the problem of learning control policies for mobile robots modeled as unknown Markov Decision Processes (MDPs)
We propose a novel accelerated model-based reinforcement learning (RL) algorithm for control objectives that is capable of learning control policies significantly faster than related approaches.
arXiv Detail & Related papers (2022-05-09T17:09:51Z) - Accelerated Policy Learning with Parallel Differentiable Simulation [59.665651562534755]
We present a differentiable simulator and a new policy learning algorithm (SHAC)
Our algorithm alleviates problems with local minima through a smooth critic function.
We show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms.
arXiv Detail & Related papers (2022-04-14T17:46:26Z) - Online Reinforcement Learning Control by Direct Heuristic Dynamic
Programming: from Time-Driven to Event-Driven [80.94390916562179]
Time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives.
It is desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise.
We show how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.
arXiv Detail & Related papers (2020-06-16T05:51:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.