Evolving Cache Schedules for Fast Diffusion Policy Inference

• URL: http://arxiv.org/abs/2607.20293v1
• Date: Wed, 22 Jul 2026 15:40:11 GMT
• Title: Evolving Cache Schedules for Fast Diffusion Policy Inference
• Abstract summary: Diffusion policies achieve strong visuomotor control by iteratively denoising action chunks, but repeated denoising makes real-time deployment computationally demanding.<n>We introduce Evolving Cache Schedules (EVO), a training-free acceleration framework that globally schedules cache refreshes via evolutionary search.<n>EVO preserves near-full performance while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G.
• Score: 6.879031540914794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion policies achieve strong visuomotor control by iteratively denoising action chunks, but repeated denoising makes real-time deployment computationally demanding. Cache-based methods reduce inference cost by reusing intermediate activations, but existing training-free schedules typically allocate computation uniformly across blocks, ignoring heterogeneous redundancy across blocks and leading to a suboptimal performance-efficiency trade-off. To bridge this gap, we introduce Evolving Cache Schedules (EVO), a training-free acceleration framework that globally schedules cache refreshes via evolutionary search. EVO represents each candidate as a complete schedule over the block-timestep lattice. Thus, redundant transformer computations during iterative denoising can be skipped through cache reuse while preserving closed-loop rollout performance. To make the search practical, EVO introduces redundancy-aware initialization, which seeds the population with promising schedules, and target-conditioned early stopping, which verifies and terminates once a desired performance target is reached. The offline-optimized schedule can be directly plugged into pretrained diffusion policies without retraining. Extensive manipulation benchmarks show that EVO preserves near-full performance while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G. Source code is available at https://github.com/pillom/EVO.

Related papers

• BAG: Budget-Aware Gating for Diffusion Caching [13.854727125627107]
  BAG (Budget-Aware Gating) is a novel caching policy that unifies global budget pacing with dynamic, instance-adaptive feature reuse.<n>We train this policy via offline-to-online schedule distillation, transferring the decision-making of offline-searched schedules into a compact online gate.
  arXiv  Detail & Related papers  (2026-08-10T07:54:28Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• Reflex: Real-Time VLA Control through Streaming Inference [12.85847066934547]
  We present textbfReflex, a framework that enables textitreal-time streaming inference for flow matching policies.<n>On LIBERO and Kinetix benchmarks, Reflex achieves a 2.58$times$ inference speedup and 50Hz stable streaming, reducing reaction latency by up to 54%.
  arXiv  Detail & Related papers  (2026-07-16T07:56:43Z)
• PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models [61.39458217121294]
  Inference in diffusion large language models (dLLMs) is computationally expensive.<n>Recent sparse attention methods for dLLMs mitigate this cost via block-sparse computation.<n>We introduce PulseCol, a periodically refreshed column-sparse attention method.
  arXiv  Detail & Related papers  (2026-05-20T07:06:54Z)
• BlockVLA: Accelerating Autoregressive VLA via Block Diffusion Finetuning [41.5997751218601]
  BlockVLA is a framework that adapts pretrained AR backbones into an efficient discrete diffusion policy.<n>We conduct extensive evaluations on the LIBERO and SimplerEnv benchmarks.<n>Our model exhibits superior training efficiency, with success rates converging substantially faster than baselines.
  arXiv  Detail & Related papers  (2026-05-13T11:37:51Z)
• Not All Frames Deserve Full Computation: Accelerating Autoregressive Video Generation via Selective Computation and Predictive Extrapolation [8.70067126225172]
  Autoregressive (AR) video diffusion models enable long-form video generation but remain expensive due to repeated multi-step denoising.<n>We present SCOPE, a training-free framework for efficient AR video diffusion.<n>On MAGI-1 and SkyReels-V2, SCOPE achieves up to 4.73x speedup while maintaining quality comparable to the original output.
  arXiv  Detail & Related papers  (2026-04-03T11:34:47Z)
• Timestep-Aware Block Masking for Efficient Diffusion Model Inference [48.764748095994996]
  Diffusion Probabilistic Models (DPMs) have achieved great success in image generation but suffer from high inference latency.<n>We propose a novel framework to optimize the computational graph of pre-trained DPMs on a per-timestep basis.
  arXiv  Detail & Related papers  (2026-03-20T13:35:03Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inference [16.54910591046794]
  Large Language Models (LLMs) demonstrate exceptional capabilities across various tasks, but their deployment is constrained by high computational and memory costs.<n>We propose a novel pruning method for PD disaggregation inference, enabling more precise and efficient block and KV Cache pruning.
  arXiv  Detail & Related papers  (2025-08-29T02:29:52Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction [72.27673320976933]
  Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding.<n>Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage.<n>We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention.
  arXiv  Detail & Related papers  (2025-08-04T16:14:03Z)
• Block-wise Adaptive Caching for Accelerating Diffusion Policy [10.641633189595302]
  Block-wise Adaptive Caching(BAC) is a method to accelerate Diffusion Policy by caching intermediate action features.<n>BAC achieves up to 3x inference speedup for free on robotic benchmarks.
  arXiv  Detail & Related papers  (2025-06-16T13:14:58Z)
• HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration [31.982294870690925]
  We develop a novel learning-based caching framework dubbed HarmoniCa.<n>It incorporates Step-Wise Denoising Training (SDT) to ensure the continuity of the denoising process.<n>Our framework achieves over $40%$ latency reduction (i.e., $2.07times$ theoretical speedup) and improved performance on PixArt-$alpha$.
  arXiv  Detail & Related papers  (2024-10-02T16:34:29Z)
• GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization [84.57695474130273]
  Gate-based or importance-based pruning methods aim to remove channels whose importance is smallest. GDP can be plugged before convolutional layers without bells and whistles, to control the on-and-off of each channel. Experiments conducted over CIFAR-10 and ImageNet datasets show that the proposed GDP achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-09-06T03:17:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.