Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

• URL: http://arxiv.org/abs/2606.20537v1
• Date: Thu, 18 Jun 2026 17:49:36 GMT
• Title: Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving
• Abstract summary: We study the opposite regime: low-latency, small-batch, on-device physical-AI serving.<n>We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state.<n>This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries.
• Score: 0.5076419064097734
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mainstream LLM serving systems reuse prefix work mainly through paged or radix key-value (KV) caches. This is highly effective for high-throughput, high-concurrency serving, but it manages only one positional fragment of execution state: the KV cache. We study the opposite regime: low-latency, small-batch, on-device physical-AI serving, where interactive LLM agents, speech systems, and robot policies repeatedly branch, reset, interrupt, and re-enter under tight responsiveness budgets. We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state at a committed boundary. FlashRT is a white-box, backend-facing kernel runtime whose evaluated NVIDIA CUDA backend runs captured graph plans over contiguous static buffers with no block-table indirection. Because the live state is a closed set of named buffers, a capsule can snapshot, restore, fork, or roll back the whole execution boundary, including KV, recurrent state, convolution state, MTP state, and metadata. This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries. On an RTX 5090, capsule restore is byte-exact at the stored-state level and token-identical under greedy decode. A KV-only ablation diverges, showing that recurrent state is load-bearing. GPU-resident snapshot and restore are sub-millisecond, and TTFT speedup over cold prefill grows from 3.9x at 2k tokens to 27x at 16k tokens. On Jetson AGX Thor and DGX Spark, the same correctness and structural properties hold. Capsules are not a replacement for high-throughput KV-cache serving; they define a complementary latency-first serving point for explicit execution-state reuse.

Related papers

• Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap [0.0]
  elastic KV cache lends the reserve to the KV pool during decode and returns it before prefill.<n>A static commit of same memory is unsafe crashing on prefill bursts, which makes the dynamic toggle necessary.<n>It only pays off if a small prefill chunk size badly hurts prefill latency.
  arXiv  Detail & Related papers  (2026-08-24T15:49:39Z)
• WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs [60.6692467625441]
  Waxing-and-Waning KV cache classifies KV-heads into anchor, tidal, and fixed roles via offline calibration.<n>On Libri-Speech with two 3B backbones, WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate.
  arXiv  Detail & Related papers  (2026-08-24T01:48:30Z)
• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• Back from the Future: Key-Value Cache Management by Counter-Causal Surprise [67.1056509495879]
  Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.<n>We propose a simple yet effective KV eviction scheme motivated by the insight that past tokens which can be well-predicted from more recent tokens are redundant.<n>We evaluate our strategy on various open-source LLMs and benchmark datasets showing competitive or improved performance over other state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-30T02:42:51Z)
• PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint [11.750747594417339]
  We present PHOENIX, a fault-tolerance mechanism that restores large language model (LLM) training via hot-swapping.<n>We show that PHOENIX simultaneously achieves both zero-overhead error-free execution and extremely low recovery cost.
  arXiv  Detail & Related papers  (2026-07-02T03:17:58Z)
• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding [11.799175681800696]
  Key-value ( KV) caching accelerates inference of large language models (LLMs) by reusing past computations for generated tokens.<n>Existing solutions either restrict reuse to prefixes or require expensive memory materialization for positional re-encoding.<n>We introduce LazyAttention, a novel attention mechanism that kernelizes deferred positional encoding to enable zero-copy, position-agnostic KV reuse.
  arXiv  Detail & Related papers  (2026-06-03T00:12:22Z)
• AURA: Action-Gated Memory for Robot Policies at Constant VRAM [0.0]
  AURA-Mem wraps a frozen vision-language-action backbone with a constant-size recurrent memory and a learned gate.<n>AURA-Mem matches the best O(1) baseline in accuracy while using 5.19-6.13 times fewer writes, and up to 9.19 times fewer writes on easier configurations.
  arXiv  Detail & Related papers  (2026-06-01T18:38:21Z)
• Runtime-Certified Bounded-Error Quantized Attention [0.0]
  We present a tiered KV cache architecture that enables runtime-certified attention.<n>A two-term error decomposition yields per-head, per-step bounds on (i) attention distribution distortion from key quantization and (ii) value reconstruction error.<n>Across PG-19, NIAH, and RULER benchmarks, the system matches dense FP16 KV quality within noise for language modelling and retrieval tasks.
  arXiv  Detail & Related papers  (2026-05-20T08:04:40Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• KV-RM: Regularizing KV-Cache Movement for Static-Graph LLM Serving [6.997204534634303]
  We present KV-RM, a runtime design that regularizes KV-cache movement beneath a static-graph LLM decoder.<n> KV-RM improves mixed-length decoding throughput and tail latency relative to a static-graph baseline.
  arXiv  Detail & Related papers  (2026-05-10T20:10:26Z)
• Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start [33.16822047715293]
  Cold-start latency remains a major bottleneck for modern LLM service providers.<n>Existing approaches either rely on brittle kernel-specific patching or executable process context.<n>We present a Foundry template-based graph materialization system that persists both graph topology and execution context.
  arXiv  Detail & Related papers  (2026-04-08T04:31:34Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.