Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning

• URL: http://arxiv.org/abs/2605.05544v1
• Date: Thu, 07 May 2026 00:48:25 GMT
• Title: Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning
• Abstract summary: We propose Adaptive Q-Chunking (AQC) for offline-to-online reinforcement learning.<n>AQC compares the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor.<n>We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic.
• Score: 7.084495564419559
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline-to-online reinforcement learning with action chunking eliminates multi-step off-policy bias and enables temporally coherent exploration, but all existing methods use a fixed chunk size across every state. This is suboptimal: near contact events the agent needs short chunks for reactive control, while during free-space motion long chunks provide better credit assignment. The natural solution is to train critics for several chunk sizes and select the best one at each state, but naive comparison of learned critic values systematically collapses to the shortest chunk due to discount-scale mismatch, and degrades to noise in low-value states. We propose Adaptive Q-Chunking (AQC), which resolves both failures by comparing the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor. This criterion converts biased wrong answers into unbiased near-random choices when no genuine signal exists, and becomes discriminative when a particular scale enables better planning. We prove theoretical bounds on the advantage selector's noise immunity and on the value dominance of adaptive chunking over any fixed chunk size. We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic, and can be applied to enhance the performance of large-scale VLA models that predict action sequences, significantly boosting performance on RoboCasa-GR1 tasks.

Related papers

• Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Models [29.586817761572373]
  Vision-language--action (VLA) models are powerful action generators for robot manipulation.<n>They are typically executed with fixed inference and replanning schedules.<n>We propose Elastic Queries Reinforcement Learning (EQRL) to make each VLA policy query elastic.
  arXiv  Detail & Related papers  (2026-06-12T12:06:41Z)
• Agile Online Model Selection: Resolving Adaptation Lag via Safeguarded Large Learning Rates [9.463718838347955]
  We propose a novel optimistic online mirror descent that utilizes safeguarded large learning rates up to $(T)$.<n>Our key technical contribution is a post-hoc penalty mechanism that dynamically monitors unstable updates and excludes learning rates incurring excessive regret.<n> Empirical evaluations on three synthetic and eleven diverse real-world datasets demonstrate that our approach reduces the adaptation lag from hundreds of rounds to a few rounds.
  arXiv  Detail & Related papers  (2026-05-26T12:18:08Z)
• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• Adaptive Action Chunking via Multi-Chunk Q Value Estimation [18.426163168939365]
  We propose Adaptive Action CHunking (ACH), a novel offline-to-online reinforcement learning algorithm.<n>ACH dynamically modulates chunk length during both training and inference.<n>Our mechanism allows the agent to select the most effective chunk length adaptively based on the current state.
  arXiv  Detail & Related papers  (2026-05-11T06:14:32Z)
• ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network [19.806833366817383]
  Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning.<n>Actor-critic methods with action chunking address this by operating over temporally extended actions.<n>We propose Adaptive Chunk Size Actor-Critic (ACSAC) to support flexible, state-dependent chunk sizes without task-specific tuning.
  arXiv  Detail & Related papers  (2026-05-10T10:00:34Z)
• Adaptive Action Chunking at Inference-time for Vision-Language-Action Models [22.3720651624276]
  Action chunking is a key technique to improve robotic manipulation abilities.<n>A large chunk size reduces the model's responsiveness to new information, while a small one increases the likelihood of mode-jumping.<n>We propose a novel Adaptive Action Chunking (AAC) strategy, which exploits action entropy as the cue to adaptively determine the chunk size.
  arXiv  Detail & Related papers  (2026-04-05T16:03:32Z)
• Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning [52.03884701766989]
  offline reinforcement learning (RL) algorithms typically impose constraints on action selection.<n>We propose a new neighborhood constraint that restricts action selection in the Bellman target to the union of neighborhoods of dataset actions.<n>We develop a simple yet effective algorithm, Adaptive Neighborhood-constrained Q learning (ANQ), to perform Q learning with target actions satisfying this constraint.
  arXiv  Detail & Related papers  (2025-11-04T13:42:05Z)
• Leave No Observation Behind: Real-time Correction for VLA Action Chunks [36.13271200613596]
  Asynchronous Action Chunk Correction (A2C2) is a lightweight real-time chunk correction head that runs every control step.<n>We show that A2C2 is an effective plug-in mechanism for deploying high-capacity chunking policies in real-time control.
  arXiv  Detail & Related papers  (2025-09-27T10:07:49Z)
• Adaptive Deadline and Batch Layered Synchronized Federated Learning [66.93447103966439]
  Federated learning (FL) enables collaborative model training across distributed edge devices while preserving data privacy, and typically operates in a round-based synchronous manner.<n>We propose ADEL-FL, a novel framework that jointly optimize per-round deadlines and user-specific batch sizes for layer-wise aggregation.
  arXiv  Detail & Related papers  (2025-05-29T19:59:18Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration [29.891468119032]
  We study how to fine-tune offline reinforcement learning (RL) pre-trained policy. We propose Q conditioned state entropy (QCSE) as intrinsic reward. We observe significant improvements with QCSE (about 13% for CQL and 8% for Cal-QL)
  arXiv  Detail & Related papers  (2023-10-07T00:02:05Z)
• Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage [100.8180383245813]
  We propose value-based algorithms for offline reinforcement learning (RL) We show an analogous result for vanilla Q-functions under a soft margin condition. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.
  arXiv  Detail & Related papers  (2023-02-05T14:22:41Z)
• Linear Stochastic Bandits over a Bit-Constrained Channel [37.01818450308119]
  We introduce a new linear bandit formulation over a bit-constrained channel. The goal of the server is to take actions based on estimates of an unknown model parameter to minimize cumulative regret. We prove that when the unknown model is $d$-dimensional, a channel capacity of $O(d)$ bits suffices to achieve order-optimal regret.
  arXiv  Detail & Related papers  (2022-03-02T15:54:03Z)
• CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
  We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference. We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms. Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
  arXiv  Detail & Related papers  (2021-10-21T06:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.