TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation

• URL: http://arxiv.org/abs/2603.07647v1
• Date: Sun, 08 Mar 2026 14:17:25 GMT
• Title: TempoFit: Plug-and-Play Layer-Wise Temporal KV Memory for Long-Horizon Vision-Language-Action Manipulation
• Abstract summary: We present TempoFit, a training-free temporal retrofit that upgrades frozen Vision-Language-Action policies through state-level memory.<n>Our key insight is that prefix attention K/V already form a model-native, content-addressable runtime state.<n>On LIBERO-LONG, TempoFit improves strong pretrained backbones by up to +4.0% average success rate while maintaining near-real-time latency.
• Score: 16.28432866472846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained Vision-Language-Action (VLA) policies have achieved strong single-step manipulation, but their inference remains largely memoryless, which is brittle in non-Markovian long-horizon settings with occlusion, state aliasing, and subtle post-action changes. Prior approaches inject history either by stacking frames, which scales visual tokens and latency while adding near-duplicate pixels, or by learning additional temporal interfaces that require (re-)training and may break the original single-frame inference graph. We present TempoFit, a training-free temporal retrofit that upgrades frozen VLAs through state-level memory. Our key insight is that prefix attention K/V already form a model-native, content-addressable runtime state; reusing them across timesteps introduces history without new tokens or trainable modules. TempoFit stores layer-wise FIFO prefix K/V at selected intermediate layers, performs parameter-free K-to-K retrieval with Frame-Gap Temporal Bias (FGTB), a fixed recency bias inspired by positional biases in NLP, to keep decisions present-dominant, and injects the retrieved context via pre-attention residual loading with norm-preserving rescaling to avoid distribution shift under frozen weights. On LIBERO-LONG, TempoFit improves strong pretrained backbones by up to +4.0% average success rate while maintaining near-real-time latency, and it transfers consistently to CALVIN and real-robot long-horizon tasks.

Related papers

• Self Gradient Forcing: Native Long Video Extrapolation [60.00513917489133]
  Self Gradient Forcing restores historical context-gradient gap.<n>Code and models will be released to advance research on autoregressive video generation.
  arXiv  Detail & Related papers  (2026-07-22T16:59:23Z)
• Surprise Forcing: What to Remember, When to Skip in Long Video Generation [65.0022589149937]
  Streaming autoregressive diffusion makes minute-scale video synthesis practical, but its bounded context and fixed denoising schedule allocate resources uniformly.<n>We introduce Surprise Forcing, a training-free framework that treats both limitations as online resource-allocation problems.<n> Experiments on VBench, VBench-Long, and VBench-2.0 show that the proposed allocation strategy improves long-horizon consistency and visual quality while retaining real-time streaming throughput.
  arXiv  Detail & Related papers  (2026-07-20T18:37:58Z)
• Adaptive Tokenisation Via Temporal Redundancy Masking And Latent Inpainting [18.71433885255431]
  Adaptive video tokenisation seeks to dynamically allocate token budgets based on the underlying visual complexity of a sequence.<n>We show that the latent space of a frozen continuous video tokeniser inherently encodes temporal redundancy that can be exploited directly.<n>We introduce a parameter-free adaptive token allocation mechanism that applies a fixed threshold to per-position temporal-L1 differences.
  arXiv  Detail & Related papers  (2026-06-04T13:31:12Z)
• SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition [7.555597360852762]
  SHARP (Sleep-based Hierarchical Accelerated Replay) is a framework that decomposes temporal learning into two complementary components.<n>Inspired by the accelerated replay observed in rodents during slow-wave sleep, SHARP incorporates offline (sleep) phases in which temporally structured memory traces are replayed in an accelerated form.
  arXiv  Detail & Related papers  (2026-05-30T13:55:02Z)
• V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models [48.80617385008755]
  Video language models (VideoLLMs) show strong capability in understanding, yet long-context inference is still dominated by massive redundant visual tokens in the prefill stage.<n>We propose V-CAST, a training-free, plug-and-play pruning policy for long-context video inference.
  arXiv  Detail & Related papers  (2026-03-29T11:53:32Z)
• MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens [34.228121359393775]
  We introduce MemRoPE, a training-free framework with two co-designed components.<n>Online RoPE Indexing caches unrotated keys and applies positional embeddings dynamically at attention time.<n>MemRoPE outperforms existing methods in temporal coherence, visual fidelity, and subject consistency across minute- to hour-scale generation.
  arXiv  Detail & Related papers  (2026-03-12T23:14:16Z)
• TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting [46.671658381710785]
  TimeSqueeze is a dynamic patching mechanism that adaptively selects patch boundaries within each sequence based on local signal complexity.<n>TimeSqueeze attains up to 20x faster convergence and 8x higher data efficiency compared to equivalent point-token baselines.
  arXiv  Detail & Related papers  (2026-03-11T22:38:13Z)
• On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention [53.22963042513293]
  Large language models (LLMs) excel at capturing global token dependencies via self-attention but face prohibitive compute and memory costs on lengthy inputs.<n>We first propose dual-state linear attention (A), a novel design that maintains two hidden states-one for preserving historical context and one for tracking recencythereby mitigating the short-range bias typical of linear-attention architectures.<n>We introduce DSLA-Serve, an online adaptive distillation framework that progressively replaces Transformer layers DSLA layers at inference time, guided by a sensitivity-based layer ordering.
  arXiv  Detail & Related papers  (2025-06-11T01:25:06Z)
• StPR: Spatiotemporal Preservation and Routing for Exemplar-Free Video Class-Incremental Learning [79.44594332189018]
  Class-Incremental Learning (CIL) seeks to develop models that continuously learn new action categories over time without previously acquired knowledge.<n>Existing approaches either rely on forgetting, raising concerns over memory and privacy, or adapt static image-based methods that neglect temporal modeling.<n>We propose a unified and exemplar-free VCIL framework that explicitly disentangles and preserves information.
  arXiv  Detail & Related papers  (2025-05-20T06:46:51Z)
• Efficient Pretraining Length Scaling [21.4715211093876]
  We present the Parallel Hidden Decoding Transformer (textitPHD-Transformer), a novel framework that enables efficient length scaling during pre-training.<n>textitPHD-Transformer achieves this through an innovative KV cache management strategy that distinguishes between original tokens and hidden decoding tokens.
  arXiv  Detail & Related papers  (2025-04-21T09:41:26Z)
• Online Dense Point Tracking with Streaming Memory [54.22820729477756]
  Dense point tracking is a challenging task requiring the continuous tracking of every point in the initial frame throughout a substantial portion of a video.<n>Recent point tracking algorithms usually depend on sliding windows for indirect information propagation from the first frame to the current one.<n>We present a lightweight and fast model with textbfStreaming memory for dense textbfPOint textbfTracking and online video processing.
  arXiv  Detail & Related papers  (2025-03-09T06:16:49Z)
• TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentation [80.13343299606146]
  We propose a Temporal LiDAR Aggregation and Distillation (TLAD) algorithm, which leverages historical priors to assign different aggregation steps for different classes. To make full use of temporal images, we design a Temporal Image Aggregation and Fusion (TIAF) module, which can greatly expand the camera FOV. We also develop a Static-Moving Switch Augmentation (SMSA) algorithm, which utilizes sufficient temporal information to enable objects to switch their motion states freely.
  arXiv  Detail & Related papers  (2024-07-13T03:00:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.