Decomposability and Parallel Computation of Multi-Agent LQR
- URL: http://arxiv.org/abs/2010.08615v2
- Date: Sun, 7 Mar 2021 23:16:28 GMT
- Title: Decomposability and Parallel Computation of Multi-Agent LQR
- Authors: Gangshan Jing, He Bai, Jemin George, Aranya Chakrabortty
- Abstract summary: We propose a parallel RL scheme for a linear regulator (LQR) design in a continuous-time linear MAS.
We show that if the MAS is homogeneous then this decomposition retains closed-loop optimality.
The proposed approach can guarantee significant speed-up in learning without any loss in the cumulative value of the LQR cost.
- Score: 19.710361049812608
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Individual agents in a multi-agent system (MAS) may have decoupled open-loop
dynamics, but a cooperative control objective usually results in coupled
closed-loop dynamics thereby making the control design computationally
expensive. The computation time becomes even higher when a learning strategy
such as reinforcement learning (RL) needs to be applied to deal with the
situation when the agents dynamics are not known. To resolve this problem, we
propose a parallel RL scheme for a linear quadratic regulator (LQR) design in a
continuous-time linear MAS. The idea is to exploit the structural properties of
two graphs embedded in the $Q$ and $R$ weighting matrices in the LQR objective
to define an orthogonal transformation that can convert the original LQR design
to multiple decoupled smaller-sized LQR designs. We show that if the MAS is
homogeneous then this decomposition retains closed-loop optimality. Conditions
for decomposability, an algorithm for constructing the transformation matrix, a
parallel RL algorithm, and robustness analysis when the design is applied to
non-homogeneous MAS are presented. Simulations show that the proposed approach
can guarantee significant speed-up in learning without any loss in the
cumulative value of the LQR cost.
Related papers
- Reinforcement learning for anisotropic p-adaptation and error estimation in high-order solvers [0.37109226820205005]
We present a novel approach to automate and optimize anisotropic p-adaptation in high-order h/p using Reinforcement Learning (RL)
We develop an offline training approach, decoupled from the main solver, which shows minimal overcost when performing simulations.
We derive an inexpensive RL-based error estimation approach that enables the quantification of local discretization errors.
arXiv Detail & Related papers (2024-07-26T17:55:23Z) - Provably Efficient CVaR RL in Low-rank MDPs [58.58570425202862]
We study risk-sensitive Reinforcement Learning (RL)
We propose a novel Upper Confidence Bound (UCB) bonus-driven algorithm to balance interplay between exploration, exploitation, and representation learning in CVaR RL.
We prove that our algorithm achieves a sample complexity of $epsilon$-optimal CVaR, where $H$ is the length of each episode, $A$ is the capacity of action space, and $d$ is the dimension of representations.
arXiv Detail & Related papers (2023-11-20T17:44:40Z) - An Efficient Algorithm for Clustered Multi-Task Compressive Sensing [60.70532293880842]
Clustered multi-task compressive sensing is a hierarchical model that solves multiple compressive sensing tasks.
The existing inference algorithm for this model is computationally expensive and does not scale well in high dimensions.
We propose a new algorithm that substantially accelerates model inference by avoiding the need to explicitly compute these covariance matrices.
arXiv Detail & Related papers (2023-09-30T15:57:14Z) - Sketching as a Tool for Understanding and Accelerating Self-attention
for Long Sequences [52.6022911513076]
Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules.
We propose Linformer and Informer to reduce the quadratic complexity to linear (modulo logarithmic factors) via low-dimensional projection and row selection.
Based on the theoretical analysis, we propose Skeinformer to accelerate self-attention and further improve the accuracy of matrix approximation to self-attention.
arXiv Detail & Related papers (2021-12-10T06:58:05Z) - Sample-Efficient Reinforcement Learning Is Feasible for Linearly
Realizable MDPs with Limited Revisiting [60.98700344526674]
Low-complexity models such as linear function representation play a pivotal role in enabling sample-efficient reinforcement learning.
In this paper, we investigate a new sampling protocol, which draws samples in an online/exploratory fashion but allows one to backtrack and revisit previous states in a controlled and infrequent manner.
We develop an algorithm tailored to this setting, achieving a sample complexity that scales practicallyly with the feature dimension, the horizon, and the inverse sub-optimality gap, but not the size of the state/action space.
arXiv Detail & Related papers (2021-05-17T17:22:07Z) - Joint Deep Reinforcement Learning and Unfolding: Beam Selection and
Precoding for mmWave Multiuser MIMO with Lens Arrays [54.43962058166702]
millimeter wave (mmWave) multiuser multiple-input multiple-output (MU-MIMO) systems with discrete lens arrays have received great attention.
In this work, we investigate the joint design of a beam precoding matrix for mmWave MU-MIMO systems with DLA.
arXiv Detail & Related papers (2021-01-05T03:55:04Z) - Reinforcement Learning of Structured Control for Linear Systems with
Unknown State Matrix [0.0]
We bring forth the ideas from reinforcement learning (RL) in conjunction with sufficient stability and performance guarantees.
A special control structure enabled by this RL framework is distributed learning control which is necessary for many large-scale cyber-physical systems.
arXiv Detail & Related papers (2020-11-02T17:04:34Z) - Reinforcement Learning Based Temporal Logic Control with Maximum
Probabilistic Satisfaction [5.337302350000984]
This paper presents a model-free reinforcement learning algorithm to synthesize a control policy.
The effectiveness of the RL-based control synthesis is demonstrated via simulation and experimental results.
arXiv Detail & Related papers (2020-10-14T03:49:16Z) - Competitive Mirror Descent [67.31015611281225]
Constrained competitive optimization involves multiple agents trying to minimize conflicting objectives, subject to constraints.
We propose competitive mirror descent (CMD): a general method for solving such problems based on first order information.
As a special case we obtain a novel competitive multiplicative weights algorithm for problems on the positive cone.
arXiv Detail & Related papers (2020-06-17T22:11:35Z) - Reduced-Dimensional Reinforcement Learning Control using Singular
Perturbation Approximations [9.136645265350284]
We present a set of model-free, reduced-dimensional reinforcement learning based optimal control designs for linear time-invariant singularly perturbed (SP) systems.
We first present a state-feedback and output-feedback based RL control design for a generic SP system with unknown state and input matrices.
We extend both designs to clustered multi-agent consensus networks, where the SP property reflects through clustering.
arXiv Detail & Related papers (2020-04-29T22:15:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.