Adaptive Inference Batching using Policy Gradients

• URL: http://arxiv.org/abs/2607.05272v1
• Date: Mon, 06 Jul 2026 16:20:19 GMT
• Title: Adaptive Inference Batching using Policy Gradients
• Abstract summary: We investigate whether reinforcement learning can learn adaptive routing policies.<n>We train REINFORCE and PPO agents on a discrete-event simulator against queuing theory and production traces.<n>Our results suggest RL concentrates in multi-resource scheduling rather than single-resource scheduling.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference serving systems must balance throughput and latency under bursty, heterogeneous workloads, yet the industry standard remains static batching policies that require manual tuning and cannot adapt to shifting traffic. We investigate whether reinforcement learning (RL) can learn adaptive batching and routing policies that outperform these heuristics, training REINFORCE and PPO agents on a discrete-event simulator validated against queuing theory and production traces (Azure Functions, BurstGPT). We formulate the problem as an MDP over queue state, request type and GPU availability, evaluating across standard Poisson traffic, extreme bursts, real-world traces and heterogeneous multi-GPU routing. Our central finding is a clear boundary condition for RL's value in systems problems. In single-GPU settings, a well-tuned static batching policy is already near-optimal under Poisson-like arrivals and RL offers only marginal gains (+0.1% to +1.0%). In multi-GPU heterogeneous routing, however, where fast and slow requests compete for shared resources, the agent discovers a workload-segregation policy that eliminates Head-of-Line blocking, yielding a 3.5x (348%) improvement over Round-Robin and a 48% improvement over the strongest heuristic baseline (Shortest-Queue), with 60% higher throughput and 25% lower latency while respecting SLA constraints. The policy generalizes to unseen bursty and real-world traffic despite training only on synthetic Poisson arrivals and an attention-augmented policy network converges roughly 20% faster than an MLP baseline. These results suggest RL's advantage over engineered heuristics concentrates in combinatorial, multi-resource decisions rather than single-resource temporal scheduling, a practical distinction for deciding where learned policies justify their engineering cost in production inference infrastructure.

Related papers

• X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching [40.58294819643394]
  Pretraining navigation diffusion policies rely on large-scale expert demonstrations.<n>This paper introduces a data-efficient diffusion RL post-training framework - GQRM.<n>By conducting distributed online RL training across heterogeneous embodiments, the resulting fine-tuned policy, X-NavDP, achieves state-of-the-art cross-embodiment visual navigation performance.
  arXiv  Detail & Related papers  (2026-07-30T17:26:12Z)
• Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models [61.74572554094633]
  We argue that ratio clipping is structurally ill-suited for flow models.<n>We propose Flow-DPPO, which replaces ratio clipping with a divergence proximal constraint.<n> Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency.
  arXiv  Detail & Related papers  (2026-06-09T15:59:57Z)
• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• LLMs Can Learn to Reason Via Off-Policy RL [17.2941334301927]
  Reinforcement learning approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO.<n>We propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL)<n>OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.
  arXiv  Detail & Related papers  (2026-02-22T22:12:51Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification [5.423004756752519]
  In Internet-of-Things systems, federated learning has advanced online reinforcement learning (RL) by enabling parallel policy training without sharing raw data.<n>We present FORLER, combining Q-ensemble aggregation on the server with actor rectification on devices.<n>The server robustly merges device Q-functions to curb policy pollution and shift heavy computation off resource-constrained hardware without compromising privacy.
  arXiv  Detail & Related papers  (2026-02-02T12:57:09Z)
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model [56.92219181993453]
  We propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix) to enable on-policyRFT methods like PPO and GRPO to leverage off-policy data.<n>ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio for efficient training; (2) KL-Convex policy constraint to balance the trade-off between stability and flexibility; (3) Policy reincarnation to achieve a seamless transition from efficient early-stage learning to steady improvement.
  arXiv  Detail & Related papers  (2025-07-09T14:29:45Z)
• Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone [72.17534881026995]
  We develop an offline and online fine-tuning approach called policy-agnostic RL (PA-RL)<n>We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm.
  arXiv  Detail & Related papers  (2024-12-09T17:28:03Z)
• Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints [82.43359506154117]
  We show that typical offline reinforcement learning methods fail to learn from data with non-uniform variability. Our method is simple, theoretically motivated, and improves performance across a wide range of offline RL problems in Atari games, navigation, and pixel-based manipulation.
  arXiv  Detail & Related papers  (2022-11-02T11:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.