Efficient Hyperparameter Optimization for LLM Reinforcement Learning

• URL: http://arxiv.org/abs/2606.03073v1
• Date: Tue, 02 Jun 2026 03:02:06 GMT
• Title: Efficient Hyperparameter Optimization for LLM Reinforcement Learning
• Abstract summary: Reinforcement learning for large language models (LLMs) is highly sensitive to hyperparameter configurations.<n>Existing multi-fidelity HPO methods remain inefficient for LLM RL due to the massive model scale and resource-intensive training cycles.<n>We propose Joint Fidelity Hyper Parameters Optimization (JF-HPO), which simultaneously adapts both model size and training budget as fidelity.<n>JF-HPO significantly improves the computational efficiency of each trial (up to 14.9 times) while achieving better or competitive predictive accuracy under the same time budget.
• Score: 9.333551082741506
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) for large language models (LLMs) is highly sensitive to hyperparameter configurations, making hyperparameter optimization (HPO) essential yet computationally expensive. Existing multi-fidelity HPO methods remain inefficient for LLM RL due to the massive model scale and resource-intensive training cycles. In this paper, we propose Joint Fidelity Hyperparameter Optimization (JF-HPO), which simultaneously adapts both model size and training budget as fidelity. JF-HPO is empowered by: (i) it leverages a small proxy model of the target LLM for efficient training and evaluation in each HPO trial; (ii) it integrates carefully designed early-stopping strategies based on training dynamics; (iii) it introduces an efficient checkpointing mechanism to eliminate redundant computations. Compared with existing HPO methods, JF-HPO significantly improves the computational efficiency of each trial (up to 14.9 times), while achieving better or competitive predictive accuracy under the same time budget. Notably, compared with utilizing hyperparameter configurations from the VeRL Recipe, JF-HPO delivers performance improvements ranging from 5.8% to 111.6%.

Related papers

• Convex Optimization for Alignment and Preference Learning on a Single GPU [52.997197698288936]
  Fine-tuning large language models to align with human preferences has driven the success of systems such as Gemini and ChatGPT.<n>Direct Preference Optimization (DPO) offers a simpler alternative but has limitations such as inconsistent ranking accuracy and high dependence on GPU resources.<n>We propose the Convex Optimization for Alignment and Preference Learning Algorithm (COALA), a novel lightweight strategy with strong theoretical guarantees.
  arXiv  Detail & Related papers  (2026-05-22T05:25:00Z)
• JoyAI-LLM Flash: Advancing Mid-Scale LLMs with Token Efficiency [56.46698214215968]
  JoyAI-LLM Flash is an efficient Mixture-of-Experts (MoE) language model designed to redefine the trade-off between strong performance and token efficiency.<n>JoyAI-LLM Flash is pretrained on a massive corpus of 20 trillion tokens and further optimized through a rigorous post-training pipeline.
  arXiv  Detail & Related papers  (2026-04-03T13:52:38Z)
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs [80.76334908639745]
  We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for large language models (LLMs)<n>QeRL addresses issues by combining NVFP4 quantization with Low-Rank Adaptation (LoRA)<n>Experiments demonstrate that QeRL delivers over 1.5 times speedup in the rollout phase.
  arXiv  Detail & Related papers  (2025-10-13T17:55:09Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• ULTHO: Ultra-Lightweight yet Efficient Hyperparameter Optimization in Deep Reinforcement Learning [50.53705050673944]
  We propose ULTHO, an ultra-lightweight yet powerful framework for fast HPO in deep RL within single runs.<n>Specifically, we formulate the HPO process as a multi-armed bandit with clustered arms (MABC) and link it directly to long-term return optimization.<n>We test ULTHO on benchmarks including ALE, Procgen, MiniGrid, and PyBullet.
  arXiv  Detail & Related papers  (2025-03-08T07:03:43Z)
• Grouped Sequential Optimization Strategy -- the Application of Hyperparameter Importance Assessment in Deep Learning [1.7778609937758323]
  We implement a novel HPO strategy called 'Sequential Grouping'<n>Our experiments, validated across six additional image classification datasets, demonstrate that incorporating hyper parameter importance assessment (HIA) can significantly accelerate HPO without compromising model performance.
  arXiv  Detail & Related papers  (2025-03-07T03:01:00Z)
• A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning [61.403275660120606]
  Reinforcement learning (RL)-based fine-tuning has emerged as a powerful approach for aligning diffusion models with black-box objectives.<n>We propose leave-one-out PPO (LOOP), a novel RL for diffusion fine-tuning method.<n>Our results demonstrate that LOOP effectively improves diffusion models on various black-box objectives, and achieves a better balance between computational efficiency and performance.
  arXiv  Detail & Related papers  (2025-03-02T13:43:53Z)
• ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning [42.33815055388433]
  ARLBench is a benchmark for hyperparameter optimization (HPO) in reinforcement learning (RL) It allows comparisons of diverse HPO approaches while being highly efficient in evaluation. ARLBench is an efficient, flexible, and future-oriented foundation for research on AutoRL.
  arXiv  Detail & Related papers  (2024-09-27T15:22:28Z)
• Efficient Transformer-based Hyper-parameter Optimization for Resource-constrained IoT Environments [9.72257571115249]
  We propose a novel approach that combines transformer architecture and actor-critic Reinforcement Learning model, TRL-HPO. The results show that TRL-HPO outperforms the classification results of these approaches by 6.8% within the same time frame. This paper identifies new avenues for improving RL-based HPO processes in resource-constrained environments.
  arXiv  Detail & Related papers  (2024-03-18T20:35:35Z)
• Multi-level Training and Bayesian Optimization for Economical Hyperparameter Optimization [12.92634461859467]
  In this paper, we develop an effective approach to reducing the total amount of required training time for Hyperparameter Optimization. We propose a truncated additive Gaussian process model to calibrate approximate performance measurements generated by light training. Based on the model, a sequential model-based algorithm is developed to generate the performance profile of the configuration space as well as find optimal ones.
  arXiv  Detail & Related papers  (2020-07-20T09:03:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.