A Test-Function Approach to Incremental Stability

• URL: http://arxiv.org/abs/2507.00695v1
• Date: Tue, 01 Jul 2025 11:46:52 GMT
• Title: A Test-Function Approach to Incremental Stability
• Abstract summary: The regularity of value functions, and their connection to incremental stability, can be understood in a way that is distinct from the traditional Lyapunov-based approach to certifying stability in control theory.
• Score: 33.44344966171865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents a novel framework for analyzing Incremental-Input-to-State Stability ($\delta$ISS) based on the idea of using rewards as "test functions." Whereas control theory traditionally deals with Lyapunov functions that satisfy a time-decrease condition, reinforcement learning (RL) value functions are constructed by exponentially decaying a Lipschitz reward function that may be non-smooth and unbounded on both sides. Thus, these RL-style value functions cannot be directly understood as Lyapunov certificates. We develop a new equivalence between a variant of incremental input-to-state stability of a closed-loop system under given a policy, and the regularity of RL-style value functions under adversarial selection of a H\"older-continuous reward function. This result highlights that the regularity of value functions, and their connection to incremental stability, can be understood in a way that is distinct from the traditional Lyapunov-based approach to certifying stability in control theory.

Related papers

• Stability of Control Lyapunov Function Guided Reinforcement Learning [18.29294248332071]
  Reinforcement learning (RL) has become the de facto method for achieving locomotion on humanoid robots in practice.<n>Recent work has attempted to merge control theoretic ideas with reinforcement learning through control guided learning.<n>This paper investigates the stability properties of optimal controllers using CLF-RL.
  arXiv  Detail & Related papers  (2026-05-03T17:27:44Z)
• Stabilizing Policy Optimization via Logits Convexity [59.242732612484474]
  We show that the convexity of the supervised fine-tuning loss with respect to model logits plays a key role in enabling stable training.<n>Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework.
  arXiv  Detail & Related papers  (2026-03-01T07:40:12Z)
• Unifying Stable Optimization and Reference Regularization in RLHF [64.16830602324345]
  This paper introduces a unified regularization approach that balances objectives of preventing reward hacking and maintaining stable policy updates.<n>Our simple yet principled alignment objective yields a weighted supervised fine-tuning loss with a superior trade-off, which demonstrably improves both alignment results and implementation complexity.
  arXiv  Detail & Related papers  (2026-02-12T03:31:19Z)
• Reinforcement Learning for Control with Probabilistic Stability Guarantee: A Finite-Sample Approach [21.52137522762386]
  We propose a probabilistic stability theorem that ensures mean square stability using only a finite number of sampled trajectories.<n>We develop an RL algorithm, L-REINFORCE, that extends the classical REINFORCE algorithm to stabilization problems.<n>The effectiveness of L-REINFORCE is demonstrated through simulations on a Cartpole task, where it outperforms the baseline in ensuring stability.
  arXiv  Detail & Related papers  (2026-02-09T06:27:04Z)
• Enhanced-FQL($λ$), an Efficient and Interpretable RL with novel Fuzzy Eligibility Traces and Segmented Experience Replay [0.0]
  Enhanced-FQL($$) achieves superior sample efficiency and reduced variance compared to n-step fuzzy TD and fuzzyA($$) baselines.<n>The framework's inherent interpretability, combined with its computational efficiency and theoretical convergence guarantees, makes it suitable for safety-critical applications.
  arXiv  Detail & Related papers  (2026-01-07T20:59:18Z)
• Zubov-Net: Adaptive Stability for Neural ODEs Reconciling Accuracy with Robustness [0.16355471507854133]
  We propose an adaptive stable learning framework named ZubovNet, which reformulates Zubov's equation into a consistency characterization.<n>We introduce a new paradigm for actively controlling the geometry of RoAs by directly optimizing PRoAs to reconcile accuracy and robustness.
  arXiv  Detail & Related papers  (2025-09-26T05:01:51Z)
• Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions [15.306107403623075]
  We study the problem of certifying the stability of closed-loop systems under control policies derived from optimal control or reinforcement learning (RL)<n>Classical Lyapunov methods require a strict step-wise decrease in the Lyapunov function but such a certificate is difficult to construct for a learned control policy.<n>We formulate an approach to learn generalized Lyapunov functions by augmenting RL value functions with neural network residual terms.
  arXiv  Detail & Related papers  (2025-05-16T07:36:40Z)
• On the stability of Lipschitz continuous control problems and its application to reinforcement learning [1.534667887016089]
  We address the crucial yet underexplored stability properties of the Hamilton--Jacobi--Bellman (HJB) equation in model-free reinforcement learning contexts. We bridge the gap between Lipschitz continuous optimal control problems and classical optimal control problems in the viscosity solutions framework.
  arXiv  Detail & Related papers  (2024-04-20T08:21:25Z)
• On the continuity and smoothness of the value function in reinforcement learning and optimal control [1.534667887016089]
  We show that the value function is always H"older continuous under relatively weak assumptions on the underlying system. We also show that non-differentiable value functions can be made differentiable by slightly "disturbing" the system.
  arXiv  Detail & Related papers  (2024-03-21T14:39:28Z)
• Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning [20.07425661382103]
  $Q$-Aided Conditional Supervised Learning combines stability of RCSL with the stitching capability of $Q$-functions. QCS adaptively integrates $Q$-aid into RCSL's loss function based on trajectory return.
  arXiv  Detail & Related papers  (2024-02-03T04:17:09Z)
• Online non-parametric likelihood-ratio estimation by Pearson-divergence functional minimization [55.98760097296213]
  We introduce a new framework for online non-parametric LRE (OLRE) for the setting where pairs of iid observations $(x_t sim p, x'_t sim q)$ are observed over time. We provide theoretical guarantees for the performance of the OLRE method along with empirical validation in synthetic experiments.
  arXiv  Detail & Related papers  (2023-11-03T13:20:11Z)
• Confidence-Conditioned Value Functions for Offline Reinforcement Learning [86.59173545987984]
  We propose a new form of Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence.
  arXiv  Detail & Related papers  (2022-12-08T23:56:47Z)
• Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularization [17.322284328945194]
  Reinforcement learning (RL) is recognized as lacking generalization and robustness under environmental perturbations. We propose a new regularizer named $textbfU$ncertainty $textbfS$et $textbfR$egularizer (USR)
  arXiv  Detail & Related papers  (2022-07-05T12:56:08Z)
• Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes [70.80716221080118]
  The paper takes a generative perspective on policy evaluation via temporal-difference (TD) learning. The OS-GPTD approach is developed to estimate the value function for a given policy by observing a sequence of state-reward pairs. To alleviate the limited expressiveness associated with a single fixed kernel, a weighted ensemble (E) of GP priors is employed to yield an alternative scheme.
  arXiv  Detail & Related papers  (2021-12-01T23:15:09Z)
• Gaussian Process-based Min-norm Stabilizing Controller for Control-Affine Systems with Uncertain Input Effects and Dynamics [90.81186513537777]
  We propose a novel compound kernel that captures the control-affine nature of the problem. We show that this resulting optimization problem is convex, and we call it Gaussian Process-based Control Lyapunov Function Second-Order Cone Program (GP-CLF-SOCP)
  arXiv  Detail & Related papers  (2020-11-14T01:27:32Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)
• Stable Reinforcement Learning with Unbounded State Space [27.053432445897016]
  We consider the problem of reinforcement learning with unbounded state space motivated by the classical problem of scheduling in a queueing network. Traditional policies as well as error metric that are designed for finite, bounded or compact state space, require infinite samples for providing meaningful performance guarantee. We propose stability as the notion of "goodness": the state dynamics under the policy should remain in a bounded region with high probability.
  arXiv  Detail & Related papers  (2020-06-08T05:00:25Z)
• Distributional Robustness and Regularization in Reinforcement Learning [62.23012916708608]
  We introduce a new regularizer for empirical value functions and show that it lower bounds the Wasserstein distributionally robust value function. It suggests using regularization as a practical tool for dealing with $textitexternal uncertainty$ in reinforcement learning.
  arXiv  Detail & Related papers  (2020-03-05T19:56:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.