Online Inference in Distributional Temporal-Difference Learning

• URL: http://arxiv.org/abs/2608.14408v1
• Date: Fri, 14 Aug 2026 15:51:50 GMT
• Title: Online Inference in Distributional Temporal-Difference Learning
• Abstract summary: We study online statistical inference for functionals of the return distribution under a fixed policy.<n>For the Polyak--Ruppert averaged estimator, we prove that its root-$T$ error converges weakly to a centered Gaussian random element in Cramér space.<n>For nonsmooth statistical functionals, we develop a local theory for the estimated return CDF over $T-1/2$-neighborhoods of finitely many thresholds.
• Score: 12.653922692549969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Markov trajectory. For the Polyak--Ruppert averaged estimator, we prove that its root-$T$ error converges weakly to a centered Gaussian random element in Cramér space. We also prove that, conditionally on the observed trajectory, the root-$T$ difference between the bootstrap and original averages converges weakly to the same Gaussian limit. These results justify bootstrap inference for smooth statistical functionals, including variance, CVaR, expected shortfall, and expectiles. For nonsmooth statistical functionals, we develop a local asymptotic theory for the estimated return CDF over $T^{-1/2}$-neighborhoods of finitely many thresholds, together with its bootstrap analogue. This theory allows us to conduct inference for nonsmooth statistical functionals characterized by CDF equations, including return quantiles.

Related papers

• The V-fold jackknife for semiparametric inference: variance estimation, confidence intervals, and simultaneous confidence bands [2.715949869811686]
  bootstrap has been a default tool for statistical inference because of its broad applicability and minimal analytic requirements.<n>We develop the $V$-fold jackknife as a computationally efficient and theoretically justified alternative for semiparametric inference.
  arXiv  Detail & Related papers  (2026-07-24T17:05:54Z)
• Statistical Efficiency and Inference of Quantile Distributional Reinforcement Learning [16.700969464418936]
  We study quantile-based distributional reinforcement learning from the perspective of statistical efficiency.<n>We establish a non-asymptotic error bound for $_m(n)$ and $_m$ under the supremum $W_infty$ metric.<n>We show that quantile-based estimators remain parametrically efficient in the infinite-dimensional limit.
  arXiv  Detail & Related papers  (2026-07-09T13:06:07Z)
• Efficient reductions from a Gaussian source with applications to statistical-computational tradeoffs [8.162867143465382]
  We leverage our technique to establish reduction-based computational lower bounds for several canonical high-dimensional statistical models under widely-believed conjectures in average-case complexity.<n>We show that computational lower bounds proved for spiked tensor PCA with Gaussian noise are universal, in that they extend to other non-Gaussian noise distributions within our class.
  arXiv  Detail & Related papers  (2025-10-08T17:16:36Z)
• Robust Estimation for Kernel Exponential Families with Smoothed Total Variation Distances [2.317910166616341]
  In statistical inference, we commonly assume that samples are independent and identically distributed from a probability distribution. In this paper, we explore the application of GAN-like estimators to a general class of statistical models.
  arXiv  Detail & Related papers  (2024-10-28T05:50:47Z)
• Online Bootstrap Inference with Nonconvex Stochastic Gradient Descent Estimator [0.0]
  In this paper, we investigate the theoretical properties of gradient descent (SGD) for statistical inference in the context of convex problems. We propose two coferential procedures which may contain multiple error minima.
  arXiv  Detail & Related papers  (2023-06-03T22:08:10Z)
• Distributional Reinforcement Learning with Dual Expectile-Quantile Regression [51.87411935256015]
  quantile regression approach to distributional RL provides flexible and effective way of learning arbitrary return distributions.<n>We show that distributional estimation guarantees vanish, and we empirically observe that the estimated distribution rapidly collapses to its mean.<n>Motivated by the efficiency of $L$-based learning, we propose to jointly learn expectiles and quantiles of the return distribution in a way that allows efficient learning.
  arXiv  Detail & Related papers  (2023-05-26T12:30:05Z)
• Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency [53.90687548731265]
  We study optimal procedures for estimating a linear functional based on observational data. For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
  arXiv  Detail & Related papers  (2023-01-16T02:57:37Z)
• Robust computation of optimal transport by $\beta$-potential regularization [79.24513412588745]
  Optimal transport (OT) has become a widely used tool in the machine learning field to measure the discrepancy between probability distributions. We propose regularizing OT with the beta-potential term associated with the so-called $beta$-divergence. We experimentally demonstrate that the transport matrix computed with our algorithm helps estimate a probability distribution robustly even in the presence of outliers.
  arXiv  Detail & Related papers  (2022-12-26T18:37:28Z)
• Covariance Estimators for the ROOT-SGD Algorithm in Online Learning [7.00422423634143]
  We develop two estimators for the algorithm covariance of ROOT-SGD. Our first estimator adopts the idea of plug-in. For each unknown component in the formula of the covariance, we substitute it with its empirical counterpart. Our second estimator is a Hessian-free estimator that overcomes the limitation.
  arXiv  Detail & Related papers  (2022-12-02T15:55:52Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression [68.66245730450915]
  We develop an improved method for debiasing predictions and estimating frequentist uncertainty for practical datasets. Our main contribution is SLOE, an estimator of the signal strength with convergence guarantees that reduces the computation time of estimation and inference by orders of magnitude.
  arXiv  Detail & Related papers  (2021-03-23T17:48:56Z)
• Distributional Reinforcement Learning via Moment Matching [54.16108052278444]
  We formulate a method that learns a finite set of statistics from each return distribution via neural networks. Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines.
  arXiv  Detail & Related papers  (2020-07-24T05:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.