Online Statistical Inference of Constant Sample-averaged Q-Learning

• URL: http://arxiv.org/abs/2603.26982v1
• Date: Fri, 27 Mar 2026 20:49:15 GMT
• Title: Online Statistical Inference of Constant Sample-averaged Q-Learning
• Abstract summary: We propose a framework to perform statistical online inference for a sample-averaged Q-learning approach.<n>We conduct experiments to perform inference on both the modified approach and its traditional counterpart, Q-learning.
• Score: 9.412266381808685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning algorithms have been widely used for decision-making tasks in various domains. However, the performance of these algorithms can be impacted by high variance and instability, particularly in environments with noise or sparse rewards. In this paper, we propose a framework to perform statistical online inference for a sample-averaged Q-learning approach. We adapt the functional central limit theorem (FCLT) for the modified algorithm under some general conditions and then construct confidence intervals for the Q-values via random scaling. We conduct experiments to perform inference on both the modified approach and its traditional counterpart, Q-learning using random scaling and report their coverage rates and confidence interval widths on two problems: a grid world problem as a simple toy example and a dynamic resource-matching problem as a real-world example for comparison between the two solution approaches.

Related papers

• A framework to evaluate the performance of Variational Quantum Algorithms [0.0]
  Variational Quantum Algorithms (VQAs) are promising methods for solving optimization problems on noisy quantum devices.<n> benchmarking VQAs is difficult due to their behavior and the lack of standardized performance criteria.<n>This work introduces a general framework for evaluating VQAs applied to Quadratic Unconstrained Binary Optimization problems.
  arXiv  Detail & Related papers  (2026-01-21T13:15:57Z)
• Online Inference of Constrained Optimization: Primal-Dual Optimality and Sequential Quadratic Programming [55.848340925419286]
  We study online statistical inference for the solutions of quadratic optimization problems with equality and inequality constraints.<n>We develop a sequential programming (SSQP) method to solve these problems, where the step direction is computed by sequentially performing an approximation of the objective and a linear approximation of the constraints.<n>We show that our method global almost moving-average convergence and exhibits local normality with an optimal primal-dual limiting matrix in the sense of Hjek and Le Cam.
  arXiv  Detail & Related papers  (2025-11-27T06:16:17Z)
• Asymptotic Analysis of Sample-averaged Q-learning [2.2374171443798034]
  This paper introduces a framework for time-varying batch-averaged Q-learning, termed sample-averaged Q-learning (SA-QL)<n>We leverage the functional central limit of the sample-averaged algorithm under mild conditions and develop a random scaling method for interval estimation.<n>This work establishes a unified theoretical foundation for sample-averaged Q-learning, providing insights into effective batch scheduling and statistical inference for RL algorithms.
  arXiv  Detail & Related papers  (2024-10-14T17:17:19Z)
• Consistent Submodular Maximization [27.266085572522847]
  maximizing monotone submodular functions under cardinality constraints is a classic optimization task with several applications in data mining and machine learning. In this paper we study this problem in a dynamic environment with consistency constraints: elements arrive in a streaming fashion and the goal is maintaining a constant approximation to the optimal solution while having a stable solution. We provide algorithms in this setting with different trade-offs between consistency and approximation quality.
  arXiv  Detail & Related papers  (2024-05-30T11:59:58Z)
• An Empirical Investigation of Value-Based Multi-objective Reinforcement Learning for Stochastic Environments [1.26404863283601]
  This paper examines the factors influencing the frequency with which value-based MORL Q-learning algorithms learn the SER-optimal policy. We highlight the critical impact of the noisy Q-value estimates issue on the stability and convergence of these algorithms.
  arXiv  Detail & Related papers  (2024-01-06T08:43:08Z)
• Addressing the issue of stochastic environments and local decision-making in multi-objective reinforcement learning [0.0]
  Multi-objective reinforcement learning (MORL) is a relatively new field which builds on conventional Reinforcement Learning (RL) This thesis focuses on what factors influence the frequency with which value-based MORL Q-learning algorithms learn the optimal policy for an environment.
  arXiv  Detail & Related papers  (2022-11-16T04:56:42Z)
• Instance-Dependent Confidence and Early Stopping for Reinforcement Learning [99.57168572237421]
  Various algorithms for reinforcement learning (RL) exhibit dramatic variation in their convergence rates as a function of problem structure. This research provides guarantees that explain textitex post the performance differences observed. A natural next step is to convert these theoretical guarantees into guidelines that are useful in practice.
  arXiv  Detail & Related papers  (2022-01-21T04:25:35Z)
• Distributed Methods with Compressed Communication for Solving Variational Inequalities, with Theoretical Guarantees [115.08148491584997]
  We present the first theoretically grounded distributed methods for solving variational inequalities and saddle point problems using compressed communication: MASHA1 and MASHA2. New algorithms support bidirectional compressions, and also can be modified for setting with batches and for federated learning with partial participation of clients.
  arXiv  Detail & Related papers  (2021-10-07T10:04:32Z)
• Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Minimax Problems [80.46370778277186]
  Large scale convex-concave minimax problems arise in numerous applications, including game theory, robust training, and training of generative adversarial networks. We develop a communication-efficient distributed extragrad algorithm, LocalAdaSient, with an adaptive learning rate suitable for solving convex-concave minimax problem in the. Server model. We demonstrate its efficacy through several experiments in both the homogeneous and heterogeneous settings.
  arXiv  Detail & Related papers  (2021-06-18T09:42:05Z)
• Cross Learning in Deep Q-Networks [82.20059754270302]
  We propose a novel cross Q-learning algorithm, aim at alleviating the well-known overestimation problem in value-based reinforcement learning methods. Our algorithm builds on double Q-learning, by maintaining a set of parallel models and estimate the Q-value based on a randomly selected network.
  arXiv  Detail & Related papers  (2020-09-29T04:58:17Z)
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning [102.78958681141577]
  We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
  arXiv  Detail & Related papers  (2020-07-09T17:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.