Generalized Fitted Q-Iteration with Clustered Data

• URL: http://arxiv.org/abs/2510.03912v1
• Date: Sat, 04 Oct 2025 19:33:14 GMT
• Title: Generalized Fitted Q-Iteration with Clustered Data
• Abstract summary: We propose a generalized fitted Q-iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning.<n>We find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.
• Score: 11.99141704716042
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper focuses on reinforcement learning (RL) with clustered data, which is commonly encountered in healthcare applications. We propose a generalized fitted Q-iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning to handle the intra-cluster correlations. Theoretically, we demonstrate (i) the optimalities of our Q-function and policy estimators when the correlation structure is correctly specified, and (ii) their consistencies when the structure is mis-specified. Empirically, through simulations and analyses of a mobile health dataset, we find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.

Related papers

• A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration [14.808143352500531]
  We develop a framework for fitted Q-iteration (FQI) on general measurable Borel spaces.<n>We prove that sequential Rademacher complexity controls Bellman-regression generalization under policy-dependent data collection.<n>These results lay the necessary foundations for the formal analysis of many modern deep RL algorithms.
  arXiv  Detail & Related papers  (2026-05-07T07:26:58Z)
• Toward Generative Quantum Utility via Correlation-Complexity Map [10.037801594045519]
  Correlation-Complexity Map for determining when real-world data distributions are structurally aligned with IQP-type quantum generative models.<n>Characterized by two complementary indicators: (i) a Quantum Correlation-Likeness Indicator (QCLI), computed from the dataset's correlation-order (Walsh-Hadamard/Fourier) power spectrum aggregated by interaction order and quantified via Jensen-Shannon divergence from an i.i.d. binomial reference; and (ii) a Classical Correlation-Complexity Indicator (CCI), defined as the fraction of total correlation not captured by the optimal Chow-Liu tree
  arXiv  Detail & Related papers  (2026-03-06T16:25:17Z)
• QCS-ADME: Quantum Circuit Search for Drug Property Prediction with Imbalanced Data and Regression Adaptation [11.101993017609246]
  We propose a novel training-free scoring mechanism to evaluate QML circuit performance on imbalanced classification and regression tasks.<n>Our mechanism demonstrates significant correlation between scoring metrics and test performance on imbalanced classification tasks.<n>This represents the first comprehensive approach to searching and evaluating QCS circuits specifically for regression applications.
  arXiv  Detail & Related papers  (2025-03-02T19:29:04Z)
• Rethinking Clustered Federated Learning in NOMA Enhanced Wireless Networks [60.09912912343705]
  This study explores the benefits of integrating the novel clustered federated learning (CFL) approach with non-independent and identically distributed (non-IID) datasets. A detailed theoretical analysis of the generalization gap that measures the degree of non-IID in the data distribution is presented. Solutions to address the challenges posed by non-IID conditions are proposed with the analysis of the properties.
  arXiv  Detail & Related papers  (2024-03-05T17:49:09Z)
• Pointer Networks with Q-Learning for Combinatorial Optimization [55.2480439325792]
  We introduce the Pointer Q-Network (PQN), a hybrid neural architecture that integrates model-free Q-value policy approximation with Pointer Networks (Ptr-Nets) Our empirical results demonstrate the efficacy of this approach, also testing the model in unstable environments.
  arXiv  Detail & Related papers  (2023-11-05T12:03:58Z)
• PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback [106.63518036538163]
  We present a novel unified bilevel optimization-based framework, textsfPARL, formulated to address the recently highlighted critical issue of policy alignment in reinforcement learning. Our framework addressed these concerns by explicitly parameterizing the distribution of the upper alignment objective (reward design) by the lower optimal variable. Our empirical results substantiate that the proposed textsfPARL can address the alignment concerns in RL by showing significant improvements.
  arXiv  Detail & Related papers  (2023-08-03T18:03:44Z)
• An Empirical Comparison of LM-based Question and Answer Generation Methods [79.31199020420827]
  Question and answer generation (QAG) consists of generating a set of question-answer pairs given a context. In this paper, we establish baselines with three different QAG methodologies that leverage sequence-to-sequence language model (LM) fine-tuning. Experiments show that an end-to-end QAG model, which is computationally light at both training and inference times, is generally robust and outperforms other more convoluted approaches.
  arXiv  Detail & Related papers  (2023-05-26T14:59:53Z)
• Deep Spectral Q-learning with Application to Mobile Health [11.736014576781903]
  We propose a deep spectral Q-learning algorithm to handle mixed frequency data. In theory, we prove that the mean return under the estimated optimal policy converges to that under the optimal one and establish its rate of convergence.
  arXiv  Detail & Related papers  (2023-01-03T01:55:17Z)
• Reinforcement Learning with Heterogeneous Data: Estimation and Inference [84.72174994749305]
  We introduce the K-Heterogeneous Markov Decision Process (K-Hetero MDP) to address sequential decision problems with population heterogeneity. We propose the Auto-Clustered Policy Evaluation (ACPE) for estimating the value of a given policy, and the Auto-Clustered Policy Iteration (ACPI) for estimating the optimal policy in a given policy class. We present simulations to support our theoretical findings, and we conduct an empirical study on the standard MIMIC-III dataset.
  arXiv  Detail & Related papers  (2022-01-31T20:58:47Z)
• Optimal regularizations for data generation with probabilistic graphical models [0.0]
  Empirically, well-chosen regularization schemes dramatically improve the quality of the inferred models. We consider the particular case of L 2 and L 1 regularizations in the Maximum A Posteriori (MAP) inference of generative pairwise graphical models.
  arXiv  Detail & Related papers  (2021-12-02T14:45:16Z)
• Encoding-dependent generalization bounds for parametrized quantum circuits [1.2599533416395765]
  We derive bounds for PQC-based models that depend explicitly on the strategy used for data-encoding. Our results facilitate the selection of optimal data-encoding strategies.
  arXiv  Detail & Related papers  (2021-06-07T18:01:38Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.