STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer

• URL: http://arxiv.org/abs/2508.10893v1
• Date: Thu, 14 Aug 2025 17:58:05 GMT
• Title: STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
• Authors: Yushi Lan, Yihang Luo, Fangzhou Hong, Shangchen Zhou, Honghua Chen, Zhaoyang Lyu, Shuai Yang, Bo Dai, Chen Change Loy, Xingang Pan,
• Abstract summary: We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem.<n>By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios.<n>Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments.
• Score: 72.88105562624838
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem. Existing state-of-the-art methods for multi-view reconstruction either depend on expensive global optimization or rely on simplistic memory mechanisms that scale poorly with sequence length. In contrast, STream3R introduces an streaming framework that processes image sequences efficiently using causal attention, inspired by advances in modern language modeling. By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios, including dynamic scenes where traditional methods often fail. Extensive experiments show that our method consistently outperforms prior work across both static and dynamic scene benchmarks. Moreover, STream3R is inherently compatible with LLM-style training infrastructure, enabling efficient large-scale pretraining and fine-tuning for various downstream 3D tasks. Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments. More details can be found in our project page: https://nirvanalan.github.io/projects/stream3r.

Related papers

• tttLRM: Test-Time Training for Long Context and Autoregressive 3D Reconstruction [47.43504457409347]
  tttLRM is a novel large 3D reconstruction model that leverages a Test-Time Training layer.<n>Our framework efficiently compresses multiple image observations into the fast weights of the TTT layer.<n>Online learning variant of our model supports progressive 3D reconstruction and refinement from streaming observations.
  arXiv  Detail & Related papers  (2026-02-23T18:59:45Z)
• S-MUSt3R: Sliding Multi-view 3D Reconstruction [17.018626984951823]
  This work proposes S-MUSt3R, a simple and efficient pipeline that extends the limits of foundation models for monocular 3D reconstruction.<n>We show that S-MUSt3R runs successfully on long RGB sequences and produces accurate and consistent 3D reconstruction.
  arXiv  Detail & Related papers  (2026-02-04T13:07:14Z)
• Review of Feed-forward 3D Reconstruction: From DUSt3R to VGGT [10.984522161856955]
  3D reconstruction is a cornerstone technology for numerous applications, including augmented/virtual reality, autonomous driving, and robotics.<n>Deep learning has catalyzed a paradigm shift in 3D reconstruction.<n>New models employ a unified deep network to jointly infer camera poses and dense geometry directly from an Unconstrained set of images in a single forward pass.
  arXiv  Detail & Related papers  (2025-07-11T09:41:54Z)
• DGS-LRM: Real-Time Deformable 3D Gaussian Reconstruction From Monocular Videos [52.46386528202226]
  We introduce the Deformable Gaussian Splats Large Reconstruction Model (DGS-LRM)<n>It is the first feed-forward method predicting deformable 3D Gaussian splats from a monocular posed video of any dynamic scene.<n>It achieves performance on par with state-of-the-art monocular video 3D tracking methods.
  arXiv  Detail & Related papers  (2025-06-11T17:59:58Z)
• StreamSplat: Towards Online Dynamic 3D Reconstruction from Uncalibrated Video Streams [14.211339652447462]
  Real-time reconstruction of dynamic 3D scenes from uncalibrated video streams is crucial for numerous real-world applications.<n>We introduce StreamSplat, the first fully feed-forward framework that transforms uncalibrated video streams of arbitrary length into dynamic 3D representations in an online manner.
  arXiv  Detail & Related papers  (2025-06-10T14:52:36Z)
• Easi3R: Estimating Disentangled Motion from DUSt3R Without Training [69.51086319339662]
  We introduce Easi3R, a simple yet efficient training-free method for 4D reconstruction.<n>Our approach applies attention adaptation during inference, eliminating the need for from-scratch pre-training or network fine-tuning.<n>Our experiments on real-world dynamic videos demonstrate that our lightweight attention adaptation significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-31T17:59:58Z)
• VideoLifter: Lifting Videos to 3D with Fast Hierarchical Stereo Alignment [54.66217340264935]
  VideoLifter is a novel video-to-3D pipeline that leverages a local-to-global strategy on a fragment basis.<n>It significantly accelerates the reconstruction process, reducing training time by over 82% while holding better visual quality than current SOTA methods.
  arXiv  Detail & Related papers  (2025-01-03T18:52:36Z)
• Wonderland: Navigating 3D Scenes from a Single Image [43.99037613068823]
  We introduce a large-scale reconstruction model that leverages latents from a video diffusion model to predict 3D Gaussian Splattings of scenes in a feed-forward manner.<n>We train the 3D reconstruction model to operate on the video latent space with a progressive learning strategy, enabling the efficient generation of high-quality, wide-scope, and generic 3D scenes.
  arXiv  Detail & Related papers  (2024-12-16T18:58:17Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion [118.74385965694694]
  We present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes.<n>By simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes.<n>We show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics.
  arXiv  Detail & Related papers  (2024-10-04T18:00:07Z)
• Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers [37.14235383028582]
  We introduce a novel approach for single-view reconstruction that efficiently generates a 3D model from a single image via feed-forward inference. Our method utilizes two transformer-based networks, namely a point decoder and a triplane decoder, to reconstruct 3D objects using a hybrid Triplane-Gaussian intermediate representation.
  arXiv  Detail & Related papers  (2023-12-14T17:18:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.