Regularized Centered Emphatic Temporal Difference Learning

• URL: http://arxiv.org/abs/2605.04100v1
• Date: Sat, 02 May 2026 10:32:24 GMT
• Title: Regularized Centered Emphatic Temporal Difference Learning
• Abstract summary: Off-policy temporal-difference (TD) learning with function approximation faces a structural tradeoff among stability, projection geometry, and variance control.<n>Emphatic TD (ETD) improves the off-policy projection geometry through follow-on emphasis, but the follow-on trace can have high variance.<n>We propose emphRegularized Emphatic Temporal-Difference Learning (RETD), which preserves the follow-on trace and regularizes only the auxiliary centering recursion.
• Score: 14.415762410365106
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Off-policy temporal-difference (TD) learning with function approximation faces a structural tradeoff among stability, projection geometry, and variance control. Emphatic TD (ETD) improves the off-policy projection geometry through follow-on emphasis, but the follow-on trace can have high variance. We revisit this tradeoff through Bellman-error centering. Although centering naturally removes a common drift term from TD errors, we show that a naive centered emphatic extension introduces an auxiliary coupling that can destroy the positive-definiteness of the ETD key matrix. We propose \emph{Regularized Emphatic Temporal-Difference Learning} (RETD), which preserves the follow-on trace and regularizes only the auxiliary centering recursion, corresponding to lifting the lower-right block of the coupled key matrix from \(1\) to \(1+c\). We derive the RETD core matrix, prove convergence under a conservative sufficient regularization condition, and evaluate the method on diagnostic linear off-policy prediction tasks. The experiments show that RETD avoids the instability of naive centered emphatic learning, preserves favorable emphatic geometry, and exhibits a robust intermediate regime for the regularization parameter \(c\) across the diagnostics.

Related papers

• Self-Supervised Cardiac Phase Detection via Single-Parameter Latent Orbits [44.45620787098836]
  Accurate identification of end-diastole (ED) and end-systole (ES) in echocardiography underpins the quantification of ventricular function.<n>Recent self-supervised methods either prescribe strict periodic trajectories or learn an unconstrained low-dimensional motion subspace from reconstruction or registration objectives.<n>We learn an effective latent orbit, significantly improves upon the previous state of the art in ED localisation and matches it in ES localisation while using a more constrained representation and fewer training epochs.
  arXiv  Detail & Related papers  (2026-09-10T14:56:16Z)
• Confusion-Geometry Rebalancing for Long-Tailed Adversarial Training [53.3522202888031]
  Adversarial training under long tailed distributions suffers from a dual imbalance: the class imbalance skews the training objective toward head classes.<n>We propose a Confusion Geometry Rebalancing method (CGRm) for long tail adversarial training, a plug in framework that leverages directed robust errors as training signals.
  arXiv  Detail & Related papers  (2026-08-10T14:54:00Z)
• TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning [39.35080737558648]
  We introduce TailLoR, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix.<n>A soft spectral penalty discourages updates aligned with dominant singular directions, reducing interference while routing fine-grained adaptation into the highly flexible, long-tail spectral coordinates.
  arXiv  Detail & Related papers  (2026-06-04T17:59:55Z)
• A Geometric View of SRC: Learning Representations for Stable Residual Inference [0.0]
  Reconstruction-based inference assigns a class by comparing class-wise reconstruction residuals.<n>Sparse Representation Classification (SRC) is a canonical instance whose reliability depends on the geometry of the learned representation.<n>We formalize residual-ordering stability through a residual margin and characterize geometric obstructions.
  arXiv  Detail & Related papers  (2026-05-28T09:33:18Z)
• Behavior-Aware Auxiliary Corrections for Off-Policy Temporal-Difference Prediction [17.99246244661535]
  This paper studies a behavior-aware replacement of the auxiliary covariance geometry in the linear prediction setting.<n>We show that the behavior-aware replacement can be highly beneficial by itself on some tasks, but that regularization is necessary for robust performance across harder settings.
  arXiv  Detail & Related papers  (2026-05-17T08:49:52Z)
• Geometry-Aware Semantic Reasoning for Training Free Video Anomaly Detection [9.337342114768942]
  Training-free video anomaly detection (VAD) has recently emerged as a scalable alternative to supervised approaches.<n>We introduce MM-VAD, a geometry-aware semantic reasoning framework for training free VAD.<n> MM-VAD reframes anomaly detection as adaptive test-time inference rather than fixed feature comparison.
  arXiv  Detail & Related papers  (2026-03-10T09:28:37Z)
• Less is More: Clustered Cross-Covariance Control for Offline RL [13.198112768636207]
  A fundamental challenge in offline reinforcement learning is distributional shift.<n>We propose partitioned buffer sampling that restricts updates to localized replay partitions.<n>We also introduce an explicit gradient-based corrective penalty that cancels the covariance induced bias within each update.
  arXiv  Detail & Related papers  (2026-01-28T16:55:04Z)
• The Procrustean Bed of Time Series: The Optimization Bias of Point-wise Loss [53.542743390809356]
  This paper aims to provide a first-principles analysis of the Expectation of Optimization Bias (EOB)<n>Our analysis reveals a fundamental paradigm paradox: the more deterministic and structured the time series, the more severe the bias by point-wise loss function.<n>We present a concrete solution that simultaneously achieves both principles via DFT or DWT.
  arXiv  Detail & Related papers  (2025-12-21T06:08:22Z)
• Geometric-Disentangelment Unlearning [106.99160454669902]
  gradient ascent on forget samples often harms retained knowledge.<n>We propose the Geometric-disment Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component.<n>Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects.
  arXiv  Detail & Related papers  (2025-11-21T09:58:25Z)
• Control-Augmented Autoregressive Diffusion for Data Assimilation [17.305296093966803]
  We introduce an amortized framework that augments pretrained ARDMs with a lightweight controller.<n>We evaluate this framework in the context of data assimilation (DA) for chaotic partial differential equations (PDEs)<n>Our approach reduces DA inference to a single forward rollout with on-the-fly corrections, avoiding expensive adjoint computations and/or optimizations during inference.
  arXiv  Detail & Related papers  (2025-10-08T04:37:32Z)
• Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading [9.147336466586017]
  Diabetic retinopathy grading is inherently ordinal and long-tailed.<n>We propose the Constrained Asymmetric Prior Wasserstein Autoencoder (CAP-WAE)<n>CAP-WAE consistently achieves state-of-the-art Quadratic Weighted Kappa, accuracy, and macro-F1.
  arXiv  Detail & Related papers  (2025-09-30T11:58:49Z)
• ERIS: An Energy-Guided Feature Disentanglement Framework for Out-of-Distribution Time Series Classification [51.07970070817353]
  An ideal time series classification (TSC) should be able to capture invariant representations.<n>Current methods are largely unguided, lacking the semantic direction required to isolate truly universal features.<n>We propose an end-to-end Energy-Regularized Information for Shift-Robustness framework to enable guided and reliable feature disentanglement.
  arXiv  Detail & Related papers  (2025-08-19T12:13:41Z)
• TAlignDiff: Automatic Tooth Alignment assisted by Diffusion-based Transformation Learning [16.84586622473531]
  Orthodontic treatment hinges on tooth alignment, which significantly affects occlusal function, facial aesthetics, and patients' quality of life.<n>We introduce a new automatic tooth alignment method named TAlignDiff, which is supported by diffusion-based transformation learning.<n>We integrate point cloud-based transformation regression and diffusion-based transformation modeling into a unified framework, allowing bidirectional feedback between geometric constraints and diffusion refinement.
  arXiv  Detail & Related papers  (2025-08-06T15:49:38Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.