On-Policy and Off-Policy Learning for Large Action Spaces

• URL: http://arxiv.org/abs/2607.28408v1
• Date: Thu, 30 Jul 2026 15:56:11 GMT
• Title: On-Policy and Off-Policy Learning for Large Action Spaces
• Abstract summary: This thesis studies policy learning in interactive systems where an agent observes a context, selects an action from a very large set, and receives partial feedback.<n>The main framework is contextual bandits, with two paradigms: on-policy learning, where the agent interacts sequentially with the environment and minimizes regret, and off-policy learning, where it learns from logged data collected by a logging policy.
• Score: 2.995087247817663
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This thesis studies policy learning in interactive systems where an agent observes a context, selects an action from a very large set, and receives partial feedback. The main framework is contextual bandits, with two paradigms: on-policy learning, where the agent interacts sequentially with the environment and minimizes regret, and off-policy learning, where it learns from logged data collected by a logging policy. In large action spaces, both settings face major challenges: inefficient exploration, sparse data coverage, high-variance importance weights, extrapolation bias, and difficult optimization landscapes. The first part develops structured Bayesian methods for on-policy learning. We introduce meTS, a mixed-effect extension of Thompson sampling, and dTS, which leverages diffusion-inspired priors to model dependencies between actions. These methods share information across actions and yield regret guarantees depending on an effective number of actions. The second part addresses off-policy learning. We propose sDM, a structured direct method based on latent variables, show that optimization error can dominate estimation error in large action spaces, and introduce concave, efficiently optimizable policy-weighted log-likelihood objectives. Finally, we develop differentiable pessimistic methods based on exponential smoothing and PAC-Bayesian bounds to control the bias-variance trade-off of regularized importance-sampling estimators.

Related papers

• Think2Go: Generative Next POI Recommendation with LLM Reasoning [57.37706163990998]
  Next Point-of-Interest (POI) recommendation task focuses on mining user behavioral preference patterns from historical check-ins to provide personalized suggestions for the next destination.<n>Existing methods rely on shallow contextual information and handcrafted feature interactions to predict the next POI.<n>We propose Think2Go, a novel generative next POI recommendation framework, which enhances the model's comprehension of SID representations and explores diverse spatial-temporal patterns via test-time computational scaling.
  arXiv  Detail & Related papers  (2026-07-31T03:52:20Z)
• Efficient Adaptive Data Acquisition via Pretrained Belief Representations [35.0706688594896]
  We introduce policy learning with belief representations (POLAR)<n>POLAR decouples representation learning from policy learning by leveraging pretrained predictive foundation models as belief-state encoders.<n>We find that POLAR outperforms state-of-the-art amortised methods across diverse tasks.
  arXiv  Detail & Related papers  (2026-06-23T21:40:47Z)
• PAWS: Preference Learning with Advantage-Weighted Segments [61.590521943273]
  We propose a segment-based preference learning method that performs policy updates directly using segment-level advantage functions.<n>Experiments on simulated robotic manipulation and locomotion tasks demonstrate that PAWS consistently outperforms existing PbRL approaches.
  arXiv  Detail & Related papers  (2026-06-10T12:00:17Z)
• Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only [70.43369087819332]
  Supervised fine-tuning (SFT) has emerged as a crucial method for aligning large language models with human-annotated demonstrations.<n>We propose Self-Rewarding PPO, a novel fine-tuning method that leverages on-policy techniques to enhance generalization performance.
  arXiv  Detail & Related papers  (2025-10-24T02:02:13Z)
• ACT-JEPA: Novel Joint-Embedding Predictive Architecture for Efficient Policy Representation Learning [90.41852663775086]
  ACT-JEPA is a novel architecture that integrates imitation learning and self-supervised learning.<n>We train a policy to predict action sequences and abstract observation sequences.<n>Our experiments show that ACT-JEPA improves the quality of representations by learning temporal environment dynamics.
  arXiv  Detail & Related papers  (2025-01-24T16:41:41Z)
• C$^{2}$INet: Realizing Incremental Trajectory Prediction with Prior-Aware Continual Causal Intervention [10.189508227447401]
  Trajectory prediction for multi-agents in complex scenarios is crucial for applications like autonomous driving. Existing methods often overlook environmental biases, which leads to poor generalization. We propose the Continual Causal Intervention (C$2$INet) method for generalizable multi-agent trajectory prediction.
  arXiv  Detail & Related papers  (2024-11-19T08:01:20Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.