STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstruction

• URL: http://arxiv.org/abs/2603.20284v1
• Date: Wed, 18 Mar 2026 06:36:46 GMT
• Title: STAC: Plug-and-Play Spatio-Temporal Aware Cache Compression for Streaming 3D Reconstruction
• Abstract summary: 3D reconstruction from streaming inputs requires both long-term temporal consistency and efficient memory usage.<n> STAC is a framework for streaming 3D reconstruction with large causal transformers.<n> experiments show that STAC achieves state-of-the-art reconstruction quality while reducing memory consumption by nearly 10x and accelerating inference by VG 4x.
• Score: 16.772404929988507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online 3D reconstruction from streaming inputs requires both long-term temporal consistency and efficient memory usage. Although causal VGGT transformers address this challenge through a key-value (KV) cache mechanism, the cache grows linearly with the stream length, creating a major memory bottleneck. Under limited memory budgets, early cache eviction significantly degrades reconstruction quality and temporal consistency. In this work, we observe that attention in causal transformers for 3D reconstruction exhibits intrinsic spatio-temporal sparsity. Based on this insight, we propose STAC, a Spatio-Temporally Aware Cache Compression framework for streaming 3D reconstruction with large causal transformers. STAC consists of three key components: (1) a Working Temporal Token Caching mechanism that preserves long-term informative tokens using decayed cumulative attention scores; (2) a Long-term Spatial Token Caching scheme that compresses spatially redundant tokens into voxel-aligned representations for memory-efficient storage; and (3) a Chunk-based Multi-frame Optimization strategy that jointly processes consecutive frames to improve temporal coherence and GPU efficiency. Extensive experiments show that STAC achieves state-of-the-art reconstruction quality while reducing memory consumption by nearly 10x and accelerating inference by 4x, substantially improving the scalability of real-time 3D reconstruction in streaming settings.

Related papers

• FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion [59.207505503284715]
  FadeMem organizes historical KV blocks into a temporal hierarchy under a fixed cache budget.<n>New history is inserted as fine-grained entries, while older adjacent entries are progressively merged.<n>Experiments show improved subject consistency, background stability, and temporal coherence over existing bounded-cache strategies.
  arXiv  Detail & Related papers  (2026-06-09T10:22:18Z)
• GHOST: Geometry-Hierarchical Online Streaming Token Eviction for Efficient 3D Reconstruction [23.110378663548335]
  Streaming 3D reconstruction from long monocular video sequences requires maintaining a keyvalue (KV) cache.<n>GHOST is a training-free KV cache management framework that exploits the model's own 3D geometry outputs to evict redundant tokens online.<n> Experiments on various benchmarks show that GHOST preserves excellent reconstruction quality while cutting the KV cache by nearly half and delivering 1.75x faster inference.
  arXiv  Detail & Related papers  (2026-05-15T11:07:33Z)
• StreamCacheVGGT: Streaming Visual Geometry Transformers with Robust Scoring and Hybrid Cache Compression [39.0869112189715]
  StreamCacheVGGT is a training-free framework that reimagines cache management.<n>It delivers superior reconstruction accuracy and long-term stability while strictly adhering to constant-cost constraints.
  arXiv  Detail & Related papers  (2026-04-16T17:12:10Z)
• XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression [20.18561757219652]
  XStreamVGGT is a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache.<n>XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$times$.
  arXiv  Detail & Related papers  (2026-02-25T11:02:02Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression [20.18561757219652]
  XStreamVGGT is a tuning-free approach that compresses the KV cache through joint pruning and quantization.<n>We show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage.
  arXiv  Detail & Related papers  (2026-01-03T14:59:50Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• Fast3Dcache: Training-free 3D Geometry Synthesis Acceleration [16.87269278147738]
  We propose Fast3Dcache, a training-free geometry-aware caching framework for 3D diffusion inference.<n>Our method achieves up to a 27.12% speed-up and a 54.8% reduction in FLOPs, with minimal degradation in geometric quality as measured by Chamfer Distance (2.48%) and F-Score (1.95%)
  arXiv  Detail & Related papers  (2025-11-27T15:13:32Z)
• Sliding Window Attention for Learned Video Compression [67.57073402826292]
  This work introduces 3D Sliding Window Attention (SWA), a patchless form of local attention.<n>Our method significantly improves rate-distortion performance, achieving Bjorntegaard Delta-rate savings of up to 18.6 %.
  arXiv  Detail & Related papers  (2025-10-04T20:11:43Z)
• LONG3R: Long Sequence Streaming 3D Reconstruction [29.79885827038617]
  Long3R is a novel model designed for streaming multi-view 3D scene reconstruction over longer sequences.<n>Our model achieves real-time processing by operating recurrently, maintaining and updating memory with each new observation.<n>Experiments demonstrate that LONG3R outperforms state-of-the-art streaming methods, particularly for longer sequences.
  arXiv  Detail & Related papers  (2025-07-24T09:55:20Z)
• QuantCache: Adaptive Importance-Guided Quantization with Hierarchical Latent and Layer Caching for Video Generation [84.91431271257437]
  Diffusion Transformers (DiTs) have emerged as a dominant architecture in video generation.<n>DiTs come with significant drawbacks, including increased computational and memory costs.<n>We propose QuantCache, a novel training-free inference acceleration framework.
  arXiv  Detail & Related papers  (2025-03-09T10:31:51Z)
• VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment [54.66217340264935]
  VideoLifter is a novel video-to-3D pipeline that leverages a local-to-global strategy on a fragment basis.<n>It significantly accelerates the reconstruction process, reducing training time by over 82% while holding better visual quality than current SOTA methods.
  arXiv  Detail & Related papers  (2025-01-03T18:52:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.