Improved Training Mechanism for Reinforcement Learning via Online Model Selection

• URL: http://arxiv.org/abs/2512.02214v1
• Date: Mon, 01 Dec 2025 21:25:46 GMT
• Title: Improved Training Mechanism for Reinforcement Learning via Online Model Selection
• Authors: Aida Afshar, Aldo Pacchiano,
• Abstract summary: We study the problem of online model selection in reinforcement learning, where the selector has access to a class of reinforcement learning agents and learns to adaptively select the agent with the right configuration.<n>Our goal is to establish the improved efficiency and performance gains achieved by integrating online model selection methods into reinforcement learning training procedures.
• Score: 25.377745631929557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the problem of online model selection in reinforcement learning, where the selector has access to a class of reinforcement learning agents and learns to adaptively select the agent with the right configuration. Our goal is to establish the improved efficiency and performance gains achieved by integrating online model selection methods into reinforcement learning training procedures. We examine the theoretical characterizations that are effective for identifying the right configuration in practice, and address three practical criteria from a theoretical perspective: 1) Efficient resource allocation, 2) Adaptation under non-stationary dynamics, and 3) Training stability across different seeds. Our theoretical results are accompanied by empirical evidence from various model selection tasks in reinforcement learning, including neural architecture selection, step-size selection, and self model selection.

Related papers

• Scaling and Transferability of Annealing Strategies in Large Language Model Training [59.443651879173025]
  We refine a predictive framework for optimizing annealing strategies under the Warmup-Steady-Decay (WSD) scheduler.<n>Our improved framework incorporates training steps, maximum learning rate, and annealing behavior, enabling more efficient optimization of learning rate schedules.<n>We validate our findings on extensive experiments using both Dense and Mixture-of-Experts (MoE) models.
  arXiv  Detail & Related papers  (2025-12-05T16:38:33Z)
• Sensitivity of Stability: Theoretical & Empirical Analysis of Replicability for Adaptive Data Selection in Transfer Learning [3.6644458768861945]
  We introduce a mathematical framework that quantifies the fundamental trade-off between adaptation effectiveness and result consistency.<n>We show that highly adaptive strategies like gradient-based and curriculum learning achieve superior task performance but suffer from high replicability failure rates.<n>We also show that source domain pretraining provides a powerful mitigation mechanism, reducing failure rates by up to 30% while preserving performance gains.
  arXiv  Detail & Related papers  (2025-08-06T21:56:56Z)
• Bridging Offline and Online Reinforcement Learning for LLMs [71.48552761763158]
  We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online.<n>Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both.
  arXiv  Detail & Related papers  (2025-06-26T17:25:49Z)
• Automation and Feature Selection Enhancement with Reinforcement Learning (RL) [0.0]
  Reinforcement learning integrated with decision tree improves feature knowledge, state representation and selection efficiency.<n>Monte Carlo-based reinforced feature selection(MCRFS), a single-agent feature selection method reduces computational burden.<n>A dual-agent RL framework is also introduced that collectively selects features and instances, capturing the interactions between them.
  arXiv  Detail & Related papers  (2025-03-15T04:30:55Z)
• Differentiable Information Enhanced Model-Based Reinforcement Learning [48.820039382764]
  Differentiable environments have heralded new possibilities for learning control policies by offering rich differentiable information.<n>Model-based reinforcement learning (MBRL) methods exhibit the potential to effectively harness the power of differentiable information for recovering the underlying physical dynamics.<n>However, this presents two primary challenges: effectively utilizing differentiable information to 1) construct models with more accurate dynamic prediction and 2) enhance the stability of policy training.
  arXiv  Detail & Related papers  (2025-03-03T04:51:40Z)
• DODT: Enhanced Online Decision Transformer Learning through Dreamer's Actor-Critic Trajectory Forecasting [37.334947053450996]
  We introduce a novel approach that combines the Dreamer algorithm's ability to generate anticipatory trajectories with the adaptive strengths of the Online Decision Transformer. Our methodology enables parallel training where Dreamer-produced trajectories enhance the contextual decision-making of the transformer.
  arXiv  Detail & Related papers  (2024-10-15T07:27:56Z)
• P3: A Policy-Driven, Pace-Adaptive, and Diversity-Promoted Framework for data pruning in LLM Training [22.61313628957683]
  This paper introduces P3, an adaptive framework aimed at optimizing the task-specific fine-tuning process through iterative data pruning. P3 consists of three key components: Policy-driven Difficulty Measurement, Pace-Adaptive Selection, and Diversity Promotion. We validate P3 on the reasoning scenarios, APPS and MATH, demonstrating significant improvements over traditional data pruning methods.
  arXiv  Detail & Related papers  (2024-08-10T12:44:49Z)
• Model-Based Reinforcement Learning with Multi-Task Offline Pretraining [59.82457030180094]
  We present a model-based RL method that learns to transfer potentially useful dynamics and action demonstrations from offline data to a novel task. The main idea is to use the world models not only as simulators for behavior learning but also as tools to measure the task relevance. We demonstrate the advantages of our approach compared with the state-of-the-art methods in Meta-World and DeepMind Control Suite.
  arXiv  Detail & Related papers  (2023-06-06T02:24:41Z)
• Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control [46.81433026280051]
  We present a self-supervised learning approach that actively models the dynamics of nonlinear robotic systems. Our approach showcases high resilience and generalization capabilities by consistently adapting to unseen flight conditions.
  arXiv  Detail & Related papers  (2022-10-23T00:45:05Z)
• DST: Dynamic Substitute Training for Data-free Black-box Attack [79.61601742693713]
  We propose a novel dynamic substitute training attack method to encourage substitute model to learn better and faster from the target model. We introduce a task-driven graph-based structure information learning constrain to improve the quality of generated training data.
  arXiv  Detail & Related papers  (2022-04-03T02:29:11Z)
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning [137.39196753245105]
  We present a new model-based reinforcement learning algorithm that learns a multi-headed dynamics model for dynamics generalization. We incorporate context learning, which encodes dynamics-specific information from past experiences into the context latent vector. Our method exhibits superior zero-shot generalization performance across a variety of control tasks, compared to state-of-the-art RL methods.
  arXiv  Detail & Related papers  (2020-10-26T03:20:42Z)
• AutoFS: Automated Feature Selection via Diversity-aware Interactive Reinforcement Learning [34.33231470225591]
  We study the problem of balancing effectiveness and efficiency in automated feature selection. Motivated by such a computational dilemma, this study is to develop a novel feature space navigation method.
  arXiv  Detail & Related papers  (2020-08-27T09:11:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.