Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction

• URL: http://arxiv.org/abs/2604.08542v1
• Date: Thu, 09 Apr 2026 17:59:50 GMT
• Title: Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction
• Abstract summary: This paper addresses the task of large-scale 3D scene reconstruction from long video sequences.<n>Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints.<n>We propose a novel neural global context representation that efficiently compresses and retains long-range scene information.
• Score: 50.5449251266956
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses the task of large-scale 3D scene reconstruction from long video sequences. Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints. However, these methods often struggle to maintain reconstruction accuracy and consistency over long sequences due to limited memory capacity and the inability to effectively capture global contextual cues. In contrast, humans can naturally exploit the global understanding of the scene to inform local perception. Motivated by this, we propose a novel neural global context representation that efficiently compresses and retains long-range scene information, enabling the model to leverage extensive contextual cues for enhanced reconstruction accuracy and consistency. The context representation is realized through a set of lightweight neural sub-networks that are rapidly adapted during test time via self-supervised objectives, which substantially increases memory capacity without incurring significant computational overhead. The experiments on multiple large-scale benchmarks, including the KITTI Odometry~\cite{Geiger2012CVPR} and Oxford Spires~\cite{tao2025spires} datasets, demonstrate the effectiveness of our approach in handling ultra-large scenes, achieving leading pose accuracy and state-of-the-art 3D reconstruction accuracy while maintaining efficiency. Code is available at https://zju3dv.github.io/scal3r.

Related papers

• Rethinking Image-to-3D Generation with Sparse Queries: Efficiency, Capacity, and Input-View Bias [70.88698584478983]
  We present SparseGen, a novel framework for efficient image-to-3D generation.<n>We model scenes with a compact sparse set of learned 3D anchor queries and a learned expansion operator.<n>Our results argue that sparse set-latent expansion is a principled, practical alternative for efficient 3D generative modeling.
  arXiv  Detail & Related papers  (2026-04-15T14:15:02Z)
• FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation [75.74617373156902]
  FreeScale is a framework that transforms limited real-world image sequences into a scalable source of high-quality training data.<n>We show FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR.<n>Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision.
  arXiv  Detail & Related papers  (2026-04-12T08:00:53Z)
• LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows [10.300202521638274]
  We introduce the Large Sparse Reconstruction Model to study how scaling transformer context windows impacts feed-forward 3D reconstruction.<n>We show that expanding the context window -- by substantially increasing the number of active object and image tokens -- remarkably narrows this gap and enables high-fidelity 3D object reconstruction and inverse rendering.
  arXiv  Detail & Related papers  (2026-04-06T21:21:12Z)
• RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations [70.83499963694238]
  RnG (Reconstruction and Generation) is a novel feed-forward Transformer that unifies reconstruction and generation.<n>It reconstructs visible geometry and generates plausible, coherent unseen geometry and appearance.<n>Our method achieves state-of-the-art performance in both generalizable 3D reconstruction and novel view generation.
  arXiv  Detail & Related papers  (2026-03-01T17:25:32Z)
• TTT3R: 3D Reconstruction as Test-Time Training [69.51086319339662]
  We revisit the 3D reconstruction foundation models from a Test-Time Training perspective.<n>We leverage the alignment confidence between the memory state and incoming observations to derive a closed-form learning rate.<n>This training-free intervention, termed TTT3R, substantially improves length generalization.
  arXiv  Detail & Related papers  (2025-09-30T17:59:51Z)
• STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer [72.88105562624838]
  We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem.<n>By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios.<n>Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments.
  arXiv  Detail & Related papers  (2025-08-14T17:58:05Z)
• SpATr: MoCap 3D Human Action Recognition based on Spiral Auto-encoder and Transformer Network [1.4732811715354455]
  We introduce a novel approach for 3D human action recognition, denoted as SpATr (Spiral Auto-encoder and Transformer Network) A lightweight auto-encoder, based on spiral convolutions, is employed to extract spatial geometrical features from each 3D mesh. The proposed method is evaluated on three prominent 3D human action datasets: Babel, MoVi, and BMLrub.
  arXiv  Detail & Related papers  (2023-06-30T11:49:00Z)
• Cross-Dimensional Refined Learning for Real-Time 3D Visual Perception from Monocular Video [2.2299983745857896]
  We present a novel real-time capable learning method that jointly perceives a 3D scene's geometry structure and semantic labels. We propose an end-to-end cross-dimensional refinement neural network (CDRNet) to extract both 3D mesh and 3D semantic labeling in real time.
  arXiv  Detail & Related papers  (2023-03-16T11:53:29Z)
• SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion [86.77318031029404]
  We propose a framework that performs scene reconstruction and semantic scene completion jointly in an incremental and real-time manner. Our framework relies on a novel neural architecture designed to process occupancy maps and leverages voxel states to accurately and efficiently fuse semantic completion with the 3D global model.
  arXiv  Detail & Related papers  (2020-10-26T15:31:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.