Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Prediction

• URL: http://arxiv.org/abs/2605.28849v1
• Date: Sat, 16 May 2026 11:33:44 GMT
• Title: Behavior-Induced Mirror-Prox Temporal-Difference Learning for Faster Off-Policy Prediction
• Abstract summary: We propose a behavior-induced Mirror-Prox temporal-difference method called STHTD-MP.<n>We show that STHTD-MP can have a smaller mean contraction factor than GTD2-MP when the behavior-induced metric improves the saddle-point geometry.
• Score: 18.894707442188405
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Gradient temporal-difference methods provide stable off-policy prediction with linear function approximation, but their practical performance is strongly affected by the geometry induced by the auxiliary-variable metric. Existing Mirror-Prox TD methods typically use the feature covariance metric, whereas hybrid TD methods suggest that behavior-policy transition information can provide a more informative update geometry. This paper proposes a behavior-induced Mirror-Prox temporal-difference method, called STHTD-MP, which replaces the covariance metric in the primal-dual saddle-point formulation with the symmetric part of the behavior-policy Bellman matrix. The method keeps a single learning rate for the primal and auxiliary variables and applies a Mirror-Prox prediction-correction step to the resulting hybrid saddle-point operator. We provide a formal convergence analysis for fixed-policy linear prediction under standard stochastic approximation assumptions: the behavior-induced metric is positive definite, the joint mean system is Hurwitz, boundedness follows from a Lyapunov argument, and the stochastic recursion converges by the ODE method. We further derive projected-oracle ergodic gap bounds and an exact mean-operator comparison with GTD2-MP based on the spectral radius of the deterministic Mirror-Prox error matrix. The analysis shows that STHTD-MP can have a smaller mean contraction factor than GTD2-MP when the behavior-induced metric improves the saddle-point geometry. Exact numerical mean-operator analysis on two-state, Random Walk, and Boyan Chain benchmarks supports this condition, while Baird's counterexample is identified as a singular boundary case where the strict assumptions fail.

Related papers

• Conditional Diffusion for Nonparametric Instrumental Variable Quantile Regression [18.254578337569264]
  This work proposes a two-stage estimator that combines conditional diffusion modeling with a kernel-parametriced conditional moment formulation.<n>We establish an excess-risk bound for the proposed estimator and derive end-to-end total variation guarantees for the conditional diffusion model.<n> Simulation studies and a real-data application demonstrate that the proposed method outperforms existing nonparametric IVQR approaches.
  arXiv  Detail & Related papers  (2026-08-08T16:01:24Z)
• Behavior-Aware Auxiliary Corrections for Off-Policy Temporal-Difference Prediction [17.99246244661535]
  This paper studies a behavior-aware replacement of the auxiliary covariance geometry in the linear prediction setting.<n>We show that the behavior-aware replacement can be highly beneficial by itself on some tasks, but that regularization is necessary for robust performance across harder settings.
  arXiv  Detail & Related papers  (2026-05-17T08:49:52Z)
• Inversion-Free Natural Gradient Descent on Riemannian Manifolds [5.556612138252781]
  This paper proposes an inversion-free natural gradient method for probability distributions with convex parameters.<n>The manifold setting offers several advantages: one can implicitly enforce parameter constraints.<n>A limited-memory variant of the algorithm with sub-quadratic storage complexity is proposed.
  arXiv  Detail & Related papers  (2026-04-03T11:08:59Z)
• Neural Optimal Transport Meets Multivariate Conformal Prediction [58.43397908730771]
  We propose a framework for conditional vectorile regression (CVQR)<n>CVQR combines neural optimal transport with quantized optimization, and apply it to predictions.
  arXiv  Detail & Related papers  (2025-09-29T19:50:19Z)
• Euclidean Distance Matrix Completion via Asymmetric Projected Gradient Descent [25.846262685970164]
  This paper proposes and analyzes a gradient-type algorithm based on Burer-Monteiro factorization.<n>It reconstructs the point set configuration from partial Euclidean distance measurements.
  arXiv  Detail & Related papers  (2025-04-28T07:13:23Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate.<n>We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Intrinsic Bayesian Cramér-Rao Bound with an Application to Covariance Matrix Estimation [49.67011673289242]
  This paper presents a new performance bound for estimation problems where the parameter to estimate lies in a smooth manifold. It induces a geometry for the parameter manifold, as well as an intrinsic notion of the estimation error measure.
  arXiv  Detail & Related papers  (2023-11-08T15:17:13Z)
• Quantifying predictive uncertainty of aphasia severity in stroke patients with sparse heteroscedastic Bayesian high-dimensional regression [47.1405366895538]
  Sparse linear regression methods for high-dimensional data commonly assume that residuals have constant variance, which can be violated in practice. This paper proposes estimating high-dimensional heteroscedastic linear regression models using a heteroscedastic partitioned empirical Bayes Expectation Conditional Maximization algorithm.
  arXiv  Detail & Related papers  (2023-09-15T22:06:29Z)
• Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize [20.376216873620763]
  We investigate the convergence of mirror descent (SMD) under in relatively smooth and smooth convex optimization. We propose a new adaptive stepsize scheme -- the mirror Polyak stepsize (mSPS)
  arXiv  Detail & Related papers  (2021-10-28T19:49:40Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Parallel Stochastic Mirror Descent for MDPs [72.75921150912556]
  We consider the problem of learning the optimal policy for infinite-horizon Markov decision processes (MDPs) Some variant of Mirror Descent is proposed for convex programming problems with Lipschitz-continuous functionals. We analyze this algorithm in a general case and obtain an estimate of the convergence rate that does not accumulate errors during the operation of the method.
  arXiv  Detail & Related papers  (2021-02-27T19:28:39Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.