Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval

• URL: http://arxiv.org/abs/2605.09644v1
• Date: Sun, 10 May 2026 16:41:54 GMT
• Title: Attention Itself Could Retrieve.RetrieveVGGT: Training-Free Long Context Streaming 3D Reconstruction via Query-Key Similarity Retrieval
• Abstract summary: Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture.<n>StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation.<n>We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem.
• Score: 86.05617542701644
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture, but the quadratic complexity of global attention prevents long context application. StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation. We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem. By retrieving a fixed number of relevant frames at each step, VGGT maintains a controllable memory budget, which is close to its training context length. Interestingly, we find that the similarity between current frame queries and cached history frame keys at the first global attention layer of VGGT is already a strong indicator of relevance, eliminating the need for additional learned scoring. To enhance information diversity similar to a recommender system, we propose Segment Sampling so that the retrieval spans distinct relevant segments rather than a single high-similarity region. We design a pose-aware spatial memory mechanism that organizes history frames according to their already estimated camera poses, enabling location-aware retrieval. Extensive experiments demonstrate that RetrieveVGGT achieves state-of-the-art performance, outperforming StreamVGGT, TTT3R, and InfiniteVGGT while maintaining constant memory usage regardless of sequence length. Code is available at https://github.com/zzctmd/RetrieveVGGT.

Related papers

• Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation [31.251444284229397]
  We propose Coverage-Maximizing Retrieval-Augmented Generation (COVRAG) for autoregressive video generation.<n>COVRAG uses pretrained 3D priors to construct a target-view coverage map as lightweight 3D memory evidence.<n>Experiments on RealEstate10K and DL3DV10K show that COVRAG improves long-horizon geometric consistency while maintaining low latency compared to baselines.
  arXiv  Detail & Related papers  (2026-06-01T16:49:58Z)
• VGGT-$Ω$ [67.24541804164818]
  VGGT-$$ substantially improves reconstruction accuracy, efficiency, and capabilities for both static and dynamic scenes.<n>We simplify VGGT's architecture by using a single dense prediction head with multi-task supervision and removing the expensive high-resolution convolutional layers.<n> VGGT-$$ achieves strong results for reconstruction of static and dynamic scenes across multiple benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T17:59:51Z)
• Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding [55.7992006853979]
  SAVEMem is a framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.<n>We present SAVEMem, a training-free dual-stage framework that brings semantic awareness into memory generation and lets the retrieval scope adapt per query.
  arXiv  Detail & Related papers  (2026-05-08T15:40:40Z)
• I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation [56.33710337846449]
  I3DM is a novel implicit 3D-aware memory mechanism for consistent video scene generation.<n>At the core of our approach is a 3D-aware memory retrieval strategy.<n>To fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module.
  arXiv  Detail & Related papers  (2026-03-24T16:45:40Z)
• S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction [57.07346645250984]
  Streaming Semantic Gaussian Splatting (S2GS) is a strictly causal, incremental 3D Gaussian semantic field framework.<n>It does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames.<n>S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks.
  arXiv  Detail & Related papers  (2026-03-15T05:48:55Z)
• FrameVGGT: Frame Evidence Rolling Memory for streaming VGGT [3.5844726281072448]
  Streaming Visual Geometry Transformers such as StreamVGGT enable strong online 3D perception but suffer from unbounded KV-cache growth.<n>We propose FrameVGGT, a frame-driven rolling explicit-memory framework that treats each frame's incremental KV contribution as a coherent evidence block.
  arXiv  Detail & Related papers  (2026-03-08T15:46:03Z)
• OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer [14.628152488797356]
  OVGGT is a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length.<n>We show that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy.
  arXiv  Detail & Related papers  (2026-03-06T06:44:17Z)
• LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory [97.14005794889134]
  We present LoGeR, a novel architecture that scales dense 3D reconstruction to extremely long sequences without post-optimization.<n>LoGeR processes video streams in chunks, leveraging strong bidirectional priors for high-fidelity intra-chunk reasoning.<n>This memory architecture enables LoGeR to be trained on sequences of 128 frames, and generalize up to thousands of frames during inference.
  arXiv  Detail & Related papers  (2026-03-03T18:55:37Z)
• Arbitrary-Scale Video Super-Resolution with Structural and Textural Priors [80.92195378575671]
  We describe a strong baseline for Arbitra-scale super-resolution (AVSR) We then introduce ST-AVSR by equipping our baseline with a multi-scale structural and textural prior computed from the pre-trained VGG network. Comprehensive experiments show that ST-AVSR significantly improves super-resolution quality, generalization ability, and inference speed over the state-of-theart.
  arXiv  Detail & Related papers  (2024-07-13T15:27:39Z)
• LiDAR-based Recurrent 3D Semantic Segmentation with Temporal Memory Alignment [0.0]
  We propose a recurrent segmentation architecture (RNN), which takes a single range image frame as input. An alignment strategy, which we call Temporal Memory Alignment, uses ego motion to temporally align the memory between consecutive frames in feature space. We demonstrate the benefits of the presented approach on two large-scale datasets and compare it to several stateof-the-art methods.
  arXiv  Detail & Related papers  (2021-03-03T09:01:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.