Stability and Sensitivity Analysis of Relative Temporal-Difference Learning: Extended Version

• URL: http://arxiv.org/abs/2603.27874v2
• Date: Tue, 07 Apr 2026 15:04:50 GMT
• Title: Stability and Sensitivity Analysis of Relative Temporal-Difference Learning: Extended Version
• Abstract summary: Relative temporal-difference (TD) learning was introduced to the slow convergence of TD methods when the discount factor approaches one.<n>This paper analyzes relative TD learning with linear function approximation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Relative temporal-difference (TD) learning was introduced to mitigate the slow convergence of TD methods when the discount factor approaches one by subtracting a baseline from the temporal-difference update. While this idea has been studied in the tabular setting, stability guarantees with function approximation remain poorly understood. This paper analyzes relative TD learning with linear function approximation. We establish stability conditions for the algorithm and show that the choice of baseline distribution plays a central role. In particular, when the baseline is chosen as the empirical distribution of the state-action process, the algorithm is stable for any non-negative baseline weight and any discount factor. We also provide a sensitivity analysis of the resulting parameter estimates, characterizing both asymptotic bias and covariance. The asymptotic covariance and asymptotic bias are shown to remain uniformly bounded as the discount factor approaches one.

Related papers

• Statistical Inference on Gradient Flows [46.05870703588429]
  We develop a theory for time-uniform statistical inference on gradient flows arising from empirical risk minimization.<n>We introduce an algorithm-aware covariance estimator that evolves jointly with the gradient flow and avoids matrix inversion, resampling, or sample splitting.<n>Our results connect optimization dynamics with statistical inference and provide practical tools for uncertainty quantification in gradient-based methods.
  arXiv  Detail & Related papers  (2026-05-31T14:22:37Z)
• Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance [10.36372217426986]
  gradient descent (SGD) is a foundational algorithm for large-scale statistical learning and optimization.<n>We develop an efficient, model-agnostic methodology for constructing confidence regions from SGD trajectories.
  arXiv  Detail & Related papers  (2026-05-25T16:18:39Z)
• Statistical Inference for Temporal Difference Learning with Linear Function Approximation [55.80276145563105]
  We investigate the statistical properties of Temporal Difference learning with Polyak-Ruppert averaging.<n>We make three theoretical contributions that improve upon the current state-of-the-art results.
  arXiv  Detail & Related papers  (2024-10-21T15:34:44Z)
• Unlearning-based Neural Interpretations [51.99182464831169]
  We show that current baselines defined using static functions are biased, fragile and manipulable.<n>We propose UNI to compute an (un)learnable, debiased and adaptive baseline by perturbing the input towards an unlearning direction of steepest ascent.
  arXiv  Detail & Related papers  (2024-10-10T16:02:39Z)
• The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise [23.71604056844816]
  One fundamental challenge in analyzing an approximation algorithm is to establish its stability.<n>We extend the celebrated Borkar-Meyn theorem for stability bounded from the Martingale difference noise setting to the Markovian noise setting.
  arXiv  Detail & Related papers  (2024-01-15T17:20:17Z)
• A Stability Principle for Learning under Non-Stationarity [1.1510009152620668]
  We develop a versatile framework for statistical learning in non-stationary environments.<n>We prove regret bounds that are minimax optimal up to logarithmic factors when the population losses are strongly convex, or Lipschitz only.<n>We evaluate the practical performance of our approach through real-data experiments on electricity demand prediction and hospital nurse staffing.
  arXiv  Detail & Related papers  (2023-10-27T17:53:53Z)
• Non-Parametric Learning of Stochastic Differential Equations with Non-asymptotic Fast Rates of Convergence [65.63201894457404]
  We propose a novel non-parametric learning paradigm for the identification of drift and diffusion coefficients of non-linear differential equations.<n>The key idea essentially consists of fitting a RKHS-based approximation of the corresponding Fokker-Planck equation to such observations.
  arXiv  Detail & Related papers  (2023-05-24T20:43:47Z)
• Data-Driven Influence Functions for Optimization-Based Causal Inference [105.5385525290466]
  We study a constructive algorithm that approximates Gateaux derivatives for statistical functionals by finite differencing. We study the case where probability distributions are not known a priori but need to be estimated from data.
  arXiv  Detail & Related papers  (2022-08-29T16:16:22Z)
• Understanding and Detecting Convergence for Stochastic Gradient Descent with Momentum [18.88380216480131]
  This paper considers gradient descent with a constant learning rate and momentum. We construct a statistical diagnostic test for convergence to the stationary phase using the inner product between successive gradients.
  arXiv  Detail & Related papers  (2020-08-27T16:24:18Z)
• On Learning Rates and Schr\"odinger Operators [105.32118775014015]
  We present a general theoretical analysis of the effect of the learning rate. We find that the learning rate tends to zero for a broad non- neural class functions.
  arXiv  Detail & Related papers  (2020-04-15T09:52:37Z)
• Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis [102.29671176698373]
  We address the problem of policy evaluation in discounted decision processes, and provide Markov-dependent guarantees on the $ell_infty$error under a generative model. We establish both and non-asymptotic versions of local minimax lower bounds for policy evaluation, thereby providing an instance-dependent baseline by which to compare algorithms.
  arXiv  Detail & Related papers  (2020-03-16T17:15:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.