InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams

• URL: http://arxiv.org/abs/2601.02281v1
• Date: Mon, 05 Jan 2026 17:11:00 GMT
• Title: InfiniteVGGT: Visual Geometry Grounded Transformer for Endless Streams
• Abstract summary: InfiniteVGGT is a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache.<n>We introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames.
• Score: 22.277387867568834
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The grand vision of enabling persistent, large-scale 3D visual geometry understanding is shackled by the irreconcilable demands of scalability and long-term stability. While offline models like VGGT achieve inspiring geometry capability, their batch-based nature renders them irrelevant for live systems. Streaming architectures, though the intended solution for live operation, have proven inadequate. Existing methods either fail to support truly infinite-horizon inputs or suffer from catastrophic drift over long sequences. We shatter this long-standing dilemma with InfiniteVGGT, a causal visual geometry transformer that operationalizes the concept of a rolling memory through a bounded yet adaptive and perpetually expressive KV cache. Capitalizing on this, we devise a training-free, attention-agnostic pruning strategy that intelligently discards obsolete information, effectively ``rolling'' the memory forward with each new frame. Fully compatible with FlashAttention, InfiniteVGGT finally alleviates the compromise, enabling infinite-horizon streaming while outperforming existing streaming methods in long-term stability. The ultimate test for such a system is its performance over a truly infinite horizon, a capability that has been impossible to rigorously validate due to the lack of extremely long-term, continuous benchmarks. To address this critical gap, we introduce the Long3D benchmark, which, for the first time, enables a rigorous evaluation of continuous 3D geometry estimation on sequences about 10,000 frames. This provides the definitive evaluation platform for future research in long-term 3D geometry understanding. Code is available at: https://github.com/AutoLab-SAI-SJTU/InfiniteVGGT

Related papers

• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors [54.51965288351254]
  We present an online 3D vision-language model that enables real-time spatial understanding from streaming video.<n>Our approach adopts an autoregressive streaming control modeling based on the LLM's next-token prediction to learn when to respond.<n>To address the scarcity of streaming 3D-language data, we further develop a scalable data generation pipeline.
  arXiv  Detail & Related papers  (2026-06-05T04:16:24Z)
• HorizonStream: Long-Horizon Attention for Streaming 3D Reconstruction [18.749241400724493]
  Existing methods often suffer from drift, jitter, or collapse on long sequences.<n>We propose HorizonStream, a long-horizon Transformer that explicitly factorizes this kernel.<n>Experiments show that HorizonStream generalizes stably to sequences exceeding 10,000 frames with constant memory and linear time.
  arXiv  Detail & Related papers  (2026-05-22T17:50:48Z)
• Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory [37.05996890779604]
  We present Mamba-VGGT, an enhanced VGGT framework capable of persistent long-range reasoning.<n>Our work provides a scalable, linear-complexity solution for geometry-grounded world modeling in extensive 3D environments.
  arXiv  Detail & Related papers  (2026-05-17T14:34:13Z)
• I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation [56.33710337846449]
  I3DM is a novel implicit 3D-aware memory mechanism for consistent video scene generation.<n>At the core of our approach is a 3D-aware memory retrieval strategy.<n>To fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module.
  arXiv  Detail & Related papers  (2026-03-24T16:45:40Z)
• OVGGT: O(1) Constant-Cost Streaming Visual Geometry Transformer [14.628152488797356]
  OVGGT is a training-free framework that bounds both memory and compute to a fixed budget regardless of sequence length.<n>We show that OVGGT processes arbitrarily long videos within a constant VRAM envelope while achieving state-of-the-art 3D geometric accuracy.
  arXiv  Detail & Related papers  (2026-03-06T06:44:17Z)
• LongStream: Long-Sequence Streaming Autoregressive Visual Geometry [18.46781332515933]
  LongStream is a novel gauge-decoupled streaming visual geometry model for metric-scale scene reconstruction.<n>It reformulates long-range extrapolation into a constant-difficulty local task.<n>It delivers stable, metric-scale reconstruction over kilometer-scale sequences at 18 FPS.
  arXiv  Detail & Related papers  (2026-02-13T18:30:51Z)
• Geometry-Aware Rotary Position Embedding for Consistent Video World Model [48.914346802616414]
  ViewRope is a geometry-aware encoding that injects camera-ray directions directly into video transformer self-attention layers.<n>Geometry-Aware Frame-Sparse Attention exploits these geometric cues to selectively attend to relevant historical frames.<n>Our results demonstrate that ViewRope substantially improves long-term consistency while reducing computational costs.
  arXiv  Detail & Related papers  (2026-02-08T08:01:16Z)
• WorldWarp: Propagating 3D Geometry with Asynchronous Video Diffusion [78.20778143251171]
  WorldWarp is a framework that couples a 3D structural anchor with a 2D generative refiner.<n>WorldWarp maintains consistency across video chunks by dynamically updating the 3D cache at every step.<n>It achieves state-of-the-art fidelity by ensuring that 3D logic guides structure while diffusion logic perfects texture.
  arXiv  Detail & Related papers  (2025-12-22T18:53:50Z)
• Endless World: Real-Time 3D-Aware Long Video Generation [57.411689597435334]
  Endless World is a real-time framework for infinite, 3D-consistent video generation.<n>We introduce a conditional autoregressive training strategy that aligns newly generated content with existing video frames.<n>Our 3D injection mechanism enforces physical plausibility and geometric consistency throughout extended sequences.
  arXiv  Detail & Related papers  (2025-12-13T19:06:12Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• FastVGGT: Training-Free Acceleration of Visual Geometry Transformer [83.67766078575782]
  VGGT is a state-of-the-art feed-forward visual geometry model.<n>We propose FastVGGT, which leverages token merging in the 3D domain through a training-free mechanism for accelerating VGGT.<n>With 1000 input images, FastVGGT achieves a 4x speedup over VGGT while mitigating error accumulation in long-sequence scenarios.
  arXiv  Detail & Related papers  (2025-09-02T17:54:21Z)
• VGGT-Long: Chunk it, Loop it, Align it -- Pushing VGGT's Limits on Kilometer-scale Long RGB Sequences [20.693979971244342]
  VGGT-Long is a system that pushes the limits of monocular 3D reconstruction to kilometer-scale, unbounded outdoor environments.<n>We evaluate our method on KITTI, and Virtual KITTI datasets.<n>Results highlight the potential of leveraging foundation models for scalable monocular 3D scene in real-world settings.
  arXiv  Detail & Related papers  (2025-07-22T10:39:04Z)
• Breaking Down Monocular Ambiguity: Exploiting Temporal Evolution for 3D Lane Detection [79.98605061363999]
  Monocular 3D lane detection aims to estimate the 3D position of lanes from frontal-view (FV) images.<n>Existing methods are constrained by the inherent ambiguity of single-frame input.<n>We propose to unlock the rich information embedded in the temporal evolution of the scene as the vehicle moves.
  arXiv  Detail & Related papers  (2025-04-29T08:10:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.