ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network

• URL: http://arxiv.org/abs/2605.11009v1
• Date: Sun, 10 May 2026 10:00:34 GMT
• Title: ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Network
• Abstract summary: Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning.<n>Actor-critic methods with action chunking address this by operating over temporally extended actions.<n>We propose Adaptive Chunk Size Actor-Critic (ACSAC) to support flexible, state-dependent chunk sizes without task-specific tuning.
• Score: 19.806833366817383
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-horizon, sparse-reward tasks pose a fundamental challenge for reinforcement learning, since single-step TD learning suffers from bootstrapping error accumulation across successive Bellman updates. Actor-critic methods with action chunking address this by operating over temporally extended actions, which reduce the effective horizon, enable fast value backups, and support temporally consistent exploration. However, existing methods rely on a fixed chunk size and therefore cannot adaptively balance reactivity against temporal consistency. A large fixed chunk size reduces responsiveness to new observations, while a small one produces incoherent motions, forcing task-specific tuning of the chunk size. To address this limitation, we propose Adaptive Chunk Size Actor-Critic (ACSAC). ACSAC leverages a causal Transformer critic to evaluate expected returns for action chunks of different sizes. At each chunk boundary, it adaptively selects the chunk size that maximizes the expected return, supporting flexible, state-dependent chunk sizes without task-specific tuning. We prove that the ACSAC Bellman operator is a contraction whose unique fixed point is the action-value function of the adaptive policy. Experiments on OGBench demonstrate that ACSAC achieves state-of-the-art performance on long-horizon, sparse-reward manipulation tasks across both offline RL and offline-to-online RL settings.

Related papers

• Act More, Decide Less: Skill-Guided Adaptive Action Chunking for Long-Horizon LLM Agents [50.85250898062218]
  Large language model (LLM) agents for long-horizon interactive tasks typically follow a ReAct-style protocol.<n>A natural alternative is to let the agent emit variable-length action chunks.<n>But naively training such policies with standard reinforcement learning fails.
  arXiv  Detail & Related papers  (2026-09-02T03:17:11Z)
• B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations [75.39816527838181]
  B-spline Policy ( BSP) is an action representation designed for accelerating robot manipulation policies.<n> BSP parameterizes actions as continuous B-spline curves defined by a set of knots and control points.
  arXiv  Detail & Related papers  (2026-07-10T17:46:32Z)
• Adaptive Action Chunking via Multi-Chunk Q Value Estimation [18.426163168939365]
  We propose Adaptive Action CHunking (ACH), a novel offline-to-online reinforcement learning algorithm.<n>ACH dynamically modulates chunk length during both training and inference.<n>Our mechanism allows the agent to select the most effective chunk length adaptively based on the current state.
  arXiv  Detail & Related papers  (2026-05-11T06:14:32Z)
• Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning [7.084495564419559]
  We propose Adaptive Q-Chunking (AQC) for offline-to-online reinforcement learning.<n>AQC compares the advantage of each chunk size relative to a per-horizon baseline, normalized by the discount factor.<n>We demonstrate that AQC achieves state-of-the-art offline and online success rates on OGBench and Robomimic.
  arXiv  Detail & Related papers  (2026-05-07T00:48:25Z)
• Adaptive Action Chunking at Inference-time for Vision-Language-Action Models [22.3720651624276]
  Action chunking is a key technique to improve robotic manipulation abilities.<n>A large chunk size reduces the model's responsiveness to new information, while a small one increases the likelihood of mode-jumping.<n>We propose a novel Adaptive Action Chunking (AAC) strategy, which exploits action entropy as the cue to adaptively determine the chunk size.
  arXiv  Detail & Related papers  (2026-04-05T16:03:32Z)
• Decoupled Q-Chunking [63.864222078287575]
  We use chunked critics to estimate the value of short action sequences ("chunks") rather than individual actions, speeding up value backup.<n>Our key insight is to decouple the chunk length of the critic from that of the policy, allowing the policy to operate over shorter action chunks.<n>This design retains the benefits of multi-step value propagation while sidestepping both the open-loop sub-optimality and the difficulty of learning action chunking policies for long action chunks.
  arXiv  Detail & Related papers  (2025-12-11T18:52:51Z)
• Training-free Context-adaptive Attention for Efficient Long Context Modeling [57.703159205740185]
  Training-free Context-adaptive Attention (TCA-Attention) is a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference.<n>TCA-Attention achieves a 2.8$times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention.
  arXiv  Detail & Related papers  (2025-12-10T01:54:57Z)
• Leave No Observation Behind: Real-time Correction for VLA Action Chunks [36.13271200613596]
  Asynchronous Action Chunk Correction (A2C2) is a lightweight real-time chunk correction head that runs every control step.<n>We show that A2C2 is an effective plug-in mechanism for deploying high-capacity chunking policies in real-time control.
  arXiv  Detail & Related papers  (2025-09-27T10:07:49Z)
• Rectified Sparse Attention [61.7702154360081]
  Efficient long-sequence generation is a critical challenge for Large Language Models.<n>We propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification.<n> Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality.
  arXiv  Detail & Related papers  (2025-06-04T16:01:48Z)
• Solving Continuous Control via Q-learning [54.05120662838286]
  We show that a simple modification of deep Q-learning largely alleviates issues with actor-critic methods. By combining bang-bang action discretization with value decomposition, framing single-agent control as cooperative multi-agent reinforcement learning (MARL), this simple critic-only approach matches performance of state-of-the-art continuous actor-critic methods.
  arXiv  Detail & Related papers  (2022-10-22T22:55:50Z)
• Band-limited Soft Actor Critic Model [15.11069042369131]
  Soft Actor Critic (SAC) algorithms show remarkable performance in complex simulated environments. We take this idea one step further by artificially bandlimiting the target critic spatial resolution. We derive the closed form solution in the linear case and show that bandlimiting reduces the interdependency between the low frequency components of the state-action value approximation.
  arXiv  Detail & Related papers  (2020-06-19T22:52:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.