Related papers: Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification

Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification

URL: http://arxiv.org/abs/2606.06053v1
Date: Thu, 04 Jun 2026 11:54:23 GMT
Title: Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Abstract summary: We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification.<n>Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail.<n>This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms with Gibbs policy updates.
Score: 70.9534986000242
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail. This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms with Gibbs policy updates. High-probability KL-regret guarantees with explicit misspecification terms are established, recovering the standard realizable KL-regularized setting as a special case.

Related papers

Toward Plasticity-Preserving KL Regularization for Capability Retention in LLM Reinforcement Learning [43.981833556858135]
We propose a conditional regularization framework that narrows the preservation constraint from the full output distribution to correctness-conditioned response distributions.<n>CoKL achieves a more favorable balance between target-task improvement and prior-capability retention than existing regularization methods.
arXiv Detail & Related papers (2026-08-03T06:10:33Z)
Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
arXiv Detail & Related papers (2026-07-29T00:19:09Z)
SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs [55.46289074417954]
reinforcement learning with verifiable rewards (RLVR) reliably improves pass@1 on reasoning tasks, yet often fails to yield comparable gains in pass@k.<n>We argue that a central structural constraint arises from reverse-KL regularization, which stabilizes training but inherently anchors the policy to the reference distribution.<n>We propose SAGE, a principled framework that enables controllable empirical support expansion by reshaping the reverse-KL anchor distribution itself.
arXiv Detail & Related papers (2026-05-15T07:42:21Z)
Probabilistic Variational Contrastive Learning [8.23660331371415]
We propose a decoder-free framework that maximizes the evidence lower bound (ELBO)<n>We model the approximate posterior $q_theta(z|x)$ as a projected normal distribution, enabling the sampling of probabilistic embeddings.
arXiv Detail & Related papers (2025-06-11T20:26:07Z)
On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning [59.11784194183928]
Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs)<n>Regularized Policy Gradient (RPG) view shows that the widely-used $k_3$ penalty is exactly the unnormalized KL.<n>RPG-REINFORCE with RPG-Style Clip improves accuracy by up to $+6$ absolute percentage points over DAPO.
arXiv Detail & Related papers (2025-05-23T06:01:21Z)
Logarithmic Regret for Online KL-Regularized Reinforcement Learning [51.113248212150964]
KL-regularization plays a pivotal role in improving efficiency of RL fine-tuning for large language models.<n>Despite its empirical advantage, the theoretical difference between KL-regularized RL and standard RL remains largely under-explored.<n>We propose an optimistic-based KL-regularized online contextual bandit algorithm, and provide a novel analysis of its regret.
arXiv Detail & Related papers (2025-02-11T11:11:05Z)
Sharp Analysis for KL-Regularized Contextual Bandits and RLHF [52.519416266840814]
Reverse-Kullback-Leibler (KL) regularization has emerged to be a predominant technique used to enhance policy optimization in reinforcement learning.<n>We show that a simple two-stage mixed sampling strategy can achieve a sample complexity with only an additive dependence on the coverage coefficient.<n>Our results provide a comprehensive understanding of the roles of KL-regularization and data coverage in RLHF, shedding light on the design of more efficient RLHF algorithms.
arXiv Detail & Related papers (2024-11-07T11:22:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.