Optimistic Online LQR via Intrinsic Rewards

• URL: http://arxiv.org/abs/2603.28938v1
• Date: Mon, 30 Mar 2026 19:16:39 GMT
• Title: Optimistic Online LQR via Intrinsic Rewards
• Abstract summary: Intrinsic Rewards LQR (IR-LQR) is an optimistic online linear quadratic regulator (LQR) algorithm.<n> IR-LQR retains the structure of a standard LQR synthesis problem by only modifying the cost function.<n>We show that IR-LQR achieves the optimal worst-case regret rate of $sqrtT$, and compare it to various state-of-the-art online LQR algorithms.
• Score: 36.89493663384693
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimism in the face of uncertainty is a popular approach to balance exploration and exploitation in reinforcement learning. Here, we consider the online linear quadratic regulator (LQR) problem, i.e., to learn the LQR corresponding to an unknown linear dynamical system by adapting the control policy online based on closed-loop data collected during operation. In this work, we propose Intrinsic Rewards LQR (IR-LQR), an optimistic online LQR algorithm that applies the idea of intrinsic rewards originating from reinforcement learning and the concept of variance regularization to promote uncertainty-driven exploration. IR-LQR retains the structure of a standard LQR synthesis problem by only modifying the cost function, resulting in an intuitively pleasing, simple, computationally cheap, and efficient algorithm. This is in contrast to existing optimistic online LQR formulations that rely on more complicated iterative search algorithms or solve computationally demanding optimization problems. We show that IR-LQR achieves the optimal worst-case regret rate of $\sqrt{T}$, and compare it to various state-of-the-art online LQR algorithms via numerical experiments carried out on an aircraft pitch angle control and an unmanned aerial vehicle example.

Related papers

• Learning the Riccati solution operator for time-varying LQR via Deep Operator Networks [13.357666564741153]
  We propose a computational framework for replacing the repeated numerical solution of differential Riccati equations in finite-horizon Linear Quadratic Regulator (LQR) problems.<n>Instead of solving a nonlinear matrix-valued differential equation for each new system instance, we construct offline an approximation of the associated solution operator.<n>The resulting model enables fast online evaluation of approximate optimal feedbacks across a wide class of systems.
  arXiv  Detail & Related papers  (2026-04-20T16:56:34Z)
• $\ abla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• Regret Analysis of Policy Optimization over Submanifolds for Linearly Constrained Online LQG [10.32831487961828]
  We study the online linear quadratic Gaussian (LQG) problem over the manifold of stabilizing controllers that are linearly constrained to impose physical conditions such as sparsity.<n>We propose the online Newton on manifold (ONM) algorithm, which generates an online controller on-the-fly based on the second-order information of the cost function sequence.
  arXiv  Detail & Related papers  (2024-03-13T14:06:18Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Robust Learning for Smoothed Online Convex Optimization with Feedback Delay [43.85262428603507]
  We propose a novel machine learning (ML) augmented online algorithm, Robustness-Constrained Learning (RCL) RCL combines untrusted ML predictions with a trusted expert online algorithm via constrained projection to robustify the ML prediction. RCL is the first ML-augmented algorithm with a provable robustness guarantee in the case of multi-step switching cost and feedback delay.
  arXiv  Detail & Related papers  (2023-10-31T00:22:55Z)
• Efficient Methods for Non-stationary Online Learning [63.268670895111654]
  We present efficient methods for optimizing dynamic regret and adaptive regret.<n>The proposed algorithms require only one gradient query and one function evaluation at each round.<n>We also study an even stronger measure, namely "interval dynamic regret", and reduce the number of projections per round from $O(log2 T)$ to $1$.
  arXiv  Detail & Related papers  (2023-09-16T07:30:12Z)
• Regret Analysis of Online LQR Control via Trajectory Prediction and Tracking: Extended Version [1.6344851071810074]
  We propose and analyze a new method for online linear quadratic regulator (LQR) control with a priori unknown time-varying cost matrices. Our novel method involves using the available cost matrices to predict the optimal trajectory, and a tracking controller to drive the system towards it. We show in simulations that our proposed method offers improved performance compared to other previously proposed online LQR methods.
  arXiv  Detail & Related papers  (2023-02-21T02:48:57Z)
• Solving Multistage Stochastic Linear Programming via Regularized Linear Decision Rules: An Application to Hydrothermal Dispatch Planning [77.34726150561087]
  We propose a novel regularization scheme for linear decision rules (LDR) based on the AdaSO (adaptive least absolute shrinkage and selection operator) Experiments show that the overfit threat is non-negligible when using the classical non-regularized LDR to solve MSLP. For the LHDP problem, our analysis highlights the following benefits of the proposed framework in comparison to the non-regularized benchmark.
  arXiv  Detail & Related papers  (2021-10-07T02:36:14Z)
• Regret Analysis of Distributed Online LQR Control for Unknown LTI Systems [8.832969171530056]
  We study the distributed online linear quadratic regulator (LQR) problem for linear time-invariant (LTI) systems with unknown dynamics. We propose a distributed variant of the online LQR algorithm where each agent computes its system estimate during an exploration stage. We prove that our proposed algorithm scales $tildeO(T2/3)$, implying the consensus of the network over time.
  arXiv  Detail & Related papers  (2021-05-15T23:02:58Z)
• Online Apprenticeship Learning [58.45089581278177]
  In Apprenticeship Learning (AL), we are given a Markov Decision Process (MDP) without access to the cost function. The goal is to find a policy that matches the expert's performance on some predefined set of cost functions. We show that the OAL problem can be effectively solved by combining two mirror descent based no-regret algorithms.
  arXiv  Detail & Related papers  (2021-02-13T12:57:51Z)
• Decomposability and Parallel Computation of Multi-Agent LQR [19.710361049812608]
  We propose a parallel RL scheme for a linear regulator (LQR) design in a continuous-time linear MAS. We show that if the MAS is homogeneous then this decomposition retains closed-loop optimality. The proposed approach can guarantee significant speed-up in learning without any loss in the cumulative value of the LQR cost.
  arXiv  Detail & Related papers  (2020-10-16T20:15:39Z)
• Learning the Linear Quadratic Regulator from Nonlinear Observations [135.66883119468707]
  We introduce a new problem setting for continuous control called the LQR with Rich Observations, or RichLQR. In our setting, the environment is summarized by a low-dimensional continuous latent state with linear dynamics and quadratic costs. Our results constitute the first provable sample complexity guarantee for continuous control with an unknown nonlinearity in the system model and general function approximation.
  arXiv  Detail & Related papers  (2020-10-08T07:02:47Z)
• Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxation [107.06364966905821]
  We study the exploration-exploitation dilemma in the linear quadratic regulator (LQR) setting. Inspired by the extended value iteration algorithm used in optimistic algorithms for finite MDPs, we propose to relax the optimistic optimization of ofulq. We show that an $epsilon$-optimistic controller can be computed efficiently by solving at most $Obig(log (1/epsilon)big)$ Riccati equations.
  arXiv  Detail & Related papers  (2020-07-13T16:30:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.