When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

• URL: http://arxiv.org/abs/2605.26418v1
• Date: Tue, 26 May 2026 01:07:42 GMT
• Title: When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control
• Abstract summary: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test.<n>We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched training budgets, and reward functions against a calibrated rule-based baseline.
• Score: 3.1555521133346685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every workload we test - so when, if ever, does DRL actually help? We study this in RLScale-Bench, a reproducible benchmark and evaluation protocol for DRL on adaptive resource control, where an agent allocates compute to a dynamic workload under cost and service-level constraints. We evaluate PPO, DQN, A2C, SAC, TD3, and DDPG under matched architectures, training budgets, and reward functions against a calibrated rule-based baseline across six workload patterns and five seeds (240 runs), instantiate the benchmark on Kubernetes Horizontal Pod Autoscaling, and probe distribution-shift generalization. Three findings challenge common assumptions: (i) the calibrated controller achieves the lowest cost on all six workloads, though it trails the best RL agents on bursty and flash traffic; (ii) discrete-action algorithms outperform continuous-action ones by one to two orders of magnitude in constraint violations due to action-space mismatch; and (iii) no single algorithm dominates across workloads, with rankings shifting by up to four positions. The bottleneck in RL-based resource control is not algorithm selection but baseline calibration, reward engineering, and realistic evaluation protocols.

Related papers

• Scaling Automatic Research Agents via World Models [42.9808933592206]
  Recent research (AutoResearch) agents show the capability to independently implement solutions and learn from the execution.<n>Behind these gains, post-training (especially RL) plays a central role.<n>World Model RL (WMRL) replaces environment execution with a world model to remove this bottleneck.
  arXiv  Detail & Related papers  (2026-08-12T20:11:25Z)
• 3SPO: State-Score-Supervised Policy Optimization for LLM Agents [25.01222941080024]
  We propose textbfState-Score-Supervised Policy Optimization (3SPO), a novel RL algorithm that performs post-step policy optimization with dynamic state score supervision.<n> Experiments on ALFWorld and WebShop with Qwen2.5-1.5B/7B-Instruct show that 3SPO consistently outperforms GRPO by $+22.6%$ on ALFWorld and $+15.6$ points on WebShop.
  arXiv  Detail & Related papers  (2026-06-08T14:26:05Z)
• Reinforcement Learning-based Control via Y-wise Affine Neural Networks: Comparative Case Studies for Chemical Processes [0.0]
  We present an efficient and practically implementable approach for the application of reinforcement learning (RL)-based control in chemical process systems.<n>We leverage a class of RL algorithms termed Y-wise Affine Neural Network (YANN)- RL.<n>By strategically initializing actor and critic networks YANN-RL algorithms provide confident and interpretable starting points within control schemes.
  arXiv  Detail & Related papers  (2026-05-20T14:07:02Z)
• Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training? [23.212983812736343]
  Agent2 RL-Bench is a benchmark for evaluating agentic RL post-training.<n>We find that supervised pipelines dominate agent-driven post-training under fixed budgets.
  arXiv  Detail & Related papers  (2026-04-12T09:35:27Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning? [76.288870982181]
  Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures.<n> reinforcement learning (RL) has broadened exploration and enabled precise multi-objective optimization in protein design.<n>We ask if RL improves sampling efficiency and, more importantly, if it reveals capabilities not captured by supervised learning.
  arXiv  Detail & Related papers  (2025-10-02T01:31:10Z)
• Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling [8.917042313344943]
  We introduce a framework, Plume, to automatically identify and balance the skewed input trace distribution in DRL training datasets. We evaluate Plume on three networking environments, including Adaptive Bitrate Streaming, Congestion Control, and Load Balancing. Plume offers superior performance in both simulation and real-world settings, across different controllers and DRL algorithms.
  arXiv  Detail & Related papers  (2023-02-24T02:09:33Z)
• Train Hard, Fight Easy: Robust Meta Reinforcement Learning [78.16589993684698]
  A major challenge of reinforcement learning (RL) in real-world applications is the variation between environments, tasks or clients. Standard MRL methods optimize the average return over tasks, but often suffer from poor results in tasks of high risk or difficulty. In this work, we define a robust MRL objective with a controlled level. The data inefficiency is addressed via the novel Robust Meta RL algorithm (RoML)
  arXiv  Detail & Related papers  (2023-01-26T14:54:39Z)
• LCRL: Certified Policy Synthesis via Logically-Constrained Reinforcement Learning [78.2286146954051]
  LCRL implements model-free Reinforcement Learning (RL) algorithms over unknown Decision Processes (MDPs) We present case studies to demonstrate the applicability, ease of use, scalability, and performance of LCRL.
  arXiv  Detail & Related papers  (2022-09-21T13:21:00Z)
• URLB: Unsupervised Reinforcement Learning Benchmark [82.36060735454647]
  We introduce the Unsupervised Reinforcement Learning Benchmark (URLB) URLB consists of two phases: reward-free pre-training and downstream task adaptation with extrinsic rewards. We provide twelve continuous control tasks from three domains for evaluation and open-source code for eight leading unsupervised RL methods.
  arXiv  Detail & Related papers  (2021-10-28T15:07:01Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.