FrameVGGT: Frame Evidence Rolling Memory for streaming VGGT

• URL: http://arxiv.org/abs/2603.07690v1
• Date: Sun, 08 Mar 2026 15:46:03 GMT
• Title: FrameVGGT: Frame Evidence Rolling Memory for streaming VGGT
• Abstract summary: Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception but suffer from unbounded KV-cache growth.<n>We propose FrameVGGT, a frame-driven rolling explicit-memory framework that treats each frame's incremental KV contribution as a coherent evidence block.
• Score: 3.5844726281072448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception but suffer from unbounded KV-cache growth, which limits deployment over long streams. We revisit bounded-memory streaming from the perspective of geometric support. In geometry-driven reasoning, memory quality depends not only on how many tokens are retained, but also on whether the retained memory still preserves sufficiently coherent local support. This suggests that token-level retention may become less suitable under fixed budgets, as it can thin the evidence available within each contributing frame and make subsequent fusion more sensitive to weakly aligned history. Motivated by this observation, we propose FrameVGGT, a frame-driven rolling explicit-memory framework that treats each frame's incremental KV contribution as a coherent evidence block. FrameVGGT summarizes each block into a compact prototype and maintains a fixed-capacity mid-term bank of complementary frame blocks under strict budgets, with an optional lightweight anchor tier for rare prolonged degradation. Across long-sequence 3D reconstruction, video depth estimation, and camera pose benchmarks, FrameVGGT achieves favorable accuracy--memory trade-offs under bounded memory, while maintaining more stable geometry over long streams.

Related papers

• Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding [64.34948094316685]
  We introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize.<n>LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T17:28:14Z)
• PermaVid: Consistent Video Generation Across Edits via Disentangled Context Memory [80.58558679163197]
  PermaVid is a novel framework built upon a multi-modal context memory that disentangles spatial context into semantic appearance and geometric structure.<n>Our method maintains strong long-term semantic and structural consistency after edits, significantly outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-06-15T09:20:32Z)
• Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation [31.251444284229397]
  We propose Coverage-Maximizing Retrieval-Augmented Generation (COVRAG) for autoregressive video generation.<n>COVRAG uses pretrained 3D priors to construct a target-view coverage map as lightweight 3D memory evidence.<n>Experiments on RealEstate10K and DL3DV10K show that COVRAG improves long-horizon geometric consistency while maintaining low latency compared to baselines.
  arXiv  Detail & Related papers  (2026-06-01T16:49:58Z)
• Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning [53.96129188781529]
  We propose a question-guided geometric memory framework for video spatial reasoning.<n>ours injects camera-conditioned geometry into visual tokens and maintains two complementary memories.<n>ours achieves state-of-the-art performance among evaluated spatial reasoning models.
  arXiv  Detail & Related papers  (2026-05-26T17:26:29Z)
• DySink: Dynamic Frame Sinks for Autoregressive Long Video Generation [51.58559206569209]
  We propose DySink, a retrieval-based framework that maintains a compact memory bank and selects visually relevant historical frames as dynamic frame sinks.<n> Experiments on minute-long videos show that DySink consistently improves dynamic degree over strong baselines while also achieving higher temporal quality.
  arXiv  Detail & Related papers  (2026-05-20T11:01:01Z)
• Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval [86.05617542701644]
  Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture.<n>StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation.<n>We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem.
  arXiv  Detail & Related papers  (2026-05-10T16:41:54Z)
• OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer [14.628152488797356]
  OVGGT is a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length.<n>We show that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy.
  arXiv  Detail & Related papers  (2026-03-06T06:44:17Z)
• LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory [97.14005794889134]
  We present LoGeR, a novel architecture that scales dense 3D reconstruction to extremely long sequences without post-optimization.<n>LoGeR processes video streams in chunks, leveraging strong bidirectional priors for high-fidelity intra-chunk reasoning.<n>This memory architecture enables LoGeR to be trained on sequences of 128 frames, and generalize up to thousands of frames during inference.
  arXiv  Detail & Related papers  (2026-03-03T18:55:37Z)
• XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression [20.18561757219652]
  XStreamVGGT is a tuning-free approach that seamlessly integrates pruning and quantization to systematically compress the KV cache.<n>XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage by 4.42$times$.
  arXiv  Detail & Related papers  (2026-02-25T11:02:02Z)
• Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization [83.406036390582]
  Quant VideoGen (QVG) is a training free KV cache quantization framework for autoregressive video diffusion models.<n>It reduces KV memory by up to 7.0 times with less than 4% end to end latency overhead.<n>It consistently outperforms existing baselines in generation quality.
  arXiv  Detail & Related papers  (2026-02-03T00:54:32Z)
• XStreamVGGT: Extremely Memory-Efficient Streaming Vision Geometry Grounded Transformer with KV Cache Compression [20.18561757219652]
  XStreamVGGT is a tuning-free approach that compresses the KV cache through joint pruning and quantization.<n>We show that XStreamVGGT achieves mostly negligible performance degradation while substantially reducing memory usage.
  arXiv  Detail & Related papers  (2026-01-03T14:59:50Z)
• OjaKV: Context-Aware Online Low-Rank KV Cache Compression with Oja's Rule [54.37983890753086]
  We introduce OjaKV, a framework that integrates a strategic hybrid storage policy with online subspace adaptation.<n>OjaKV preserves crucial first and most recent tokens in full-rank, maintaining high-fidelity anchors for attention.<n>It applies low-rank compression by incrementally adapting the projection basis using Oja's algorithm for online principal component analysis.
  arXiv  Detail & Related papers  (2025-09-25T21:42:27Z)
• Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models [63.99949971803903]
  We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation.<n>FramePack compresses input frame contexts with frame-wise importance so that more frames can be encoded within a fixed context length.<n>We show that existing video diffusion models can be finetuned with FramePack, and analyze the differences between different packing schedules.
  arXiv  Detail & Related papers  (2025-04-17T04:02:31Z)
• Generative Inbetweening through Frame-wise Conditions-Driven Video Generation [63.43583844248389]
  generative inbetweening aims to generate intermediate frame sequences by utilizing two key frames as input.<n>We propose a Frame-wise Conditions-driven Video Generation (FCVG) method that significantly enhances the temporal stability of interpolated video frames.<n>Our FCVG demonstrates the capability to generate temporally stable videos using both linear and non-linear curves.
  arXiv  Detail & Related papers  (2024-12-16T13:19:41Z)
• Recurrent Video Restoration Transformer with Guided Deformable Attention [116.1684355529431]
  We propose RVRT, which processes local neighboring frames in parallel within a globally recurrent framework. RVRT achieves state-of-the-art performance on benchmark datasets with balanced model size, testing memory and runtime.
  arXiv  Detail & Related papers  (2022-06-05T10:36:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.