Déjà View: Looping Transformers for Multi-View 3D Reconstruction

• URL: http://arxiv.org/abs/2605.30215v2
• Date: Fri, 29 May 2026 15:11:53 GMT
• Title: Déjà View: Looping Transformers for Multi-View 3D Reconstruction
• Abstract summary: Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters.<n>We show that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture.<n>Our model, DéjView, applies a single looped transformer block recurrently to per-view features for K refinement steps.
• Score: 109.98721386523641
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters, following the broader trend of increasing model capacity in computer vision. Yet emerging evidence suggests that contiguous transformer layers often behave like repeated applications of similar operations, and multi-view reconstruction transformers refine their predictions progressively across decoder depth. We posit that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture. Our model, DéjàView, applies a single looped transformer block recurrently to per-view features for K refinement steps. Trained once, it exposes K as an inference-time compute knob, matching or outperforming substantially larger feed-forward baselines across five reconstruction benchmarks spanning indoor, outdoor, object-centric, and driving scenes, while using a fraction of their parameters and comparable or lower compute. Importantly, the same looped block formulation outperforms an otherwise identical variant with independent per-step parameters under matched training data and compute, suggesting that explicit iteration is not merely a compute-efficient substitute for capacity but a stronger inductive bias for multi-view 3D reconstruction.

Related papers

• BA-T: An Iterative Transformer for Two-View Bundle Adjustment [53.16990349592508]
  We propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space.<n> experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations.<n>BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture.
  arXiv  Detail & Related papers  (2026-06-02T07:51:14Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Structured Multidimensional Representation Learning for Large Language Models [0.0]
  Transformer architectures achieve state-of-the-art performance across a wide range of pattern recognition and natural language processing tasks.<n>We introduce a structured spectral factorization of the embedding space based on the L-product for third-order tensors.<n>We prove that the proposed L-Transformer is spectrally equivalent to p parallel Transformers operating on reduced embeddings.
  arXiv  Detail & Related papers  (2026-03-05T22:34:45Z)
• Rethinking Vision Transformer Depth via Structural Reparameterization [16.12815682992294]
  We propose a branch-based structural reparameterization technique that operates during the training phase.<n>Our approach leverages parallel branches within transformer blocks that can be systematically consolidated into streamlined single-path models.<n>When applied to ViT-Tiny, the framework successfully reduces the original 12-layer architecture to 6, 4, or as few as 3 layers while maintaining classification accuracy on ImageNet-1K.
  arXiv  Detail & Related papers  (2025-11-24T21:28:55Z)
• Understanding Multi-View Transformers [18.573401296925844]
  Multi-view transformers such as DUSt3R are revolutionizing 3D vision by solving 3D tasks in a feed-forward manner.<n>Here, we present an approach for probing and visualizing 3D representations from the residual connections of the multi-view transformers' layers.
  arXiv  Detail & Related papers  (2025-10-28T19:19:35Z)
• STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer [72.88105562624838]
  We present STream3R, a novel approach to 3D reconstruction that reformulates pointmap prediction as a decoder-only Transformer problem.<n>By learning geometric priors from large-scale 3D datasets, STream3R generalizes well to diverse and challenging scenarios.<n>Our results underscore the potential of causal Transformer models for online 3D perception, paving the way for real-time 3D understanding in streaming environments.
  arXiv  Detail & Related papers  (2025-08-14T17:58:05Z)
• PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
  Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences. We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
  arXiv  Detail & Related papers  (2024-08-20T01:56:07Z)
• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [62.40166958002558]
  We propose iTransformer, which simply applies the attention and feed-forward network on the inverted dimensions. The iTransformer model achieves state-of-the-art on challenging real-world datasets.
  arXiv  Detail & Related papers  (2023-10-10T13:44:09Z)
• Multi-view 3D Reconstruction with Transformer [34.756336770583154]
  We reformulate the multi-view 3D reconstruction as a sequence-to-sequence prediction problem. We propose a new framework named 3D Volume Transformer (VolT) for such a task. Our method achieves a new state-of-the-art accuracy in multi-view reconstruction with fewer parameters.
  arXiv  Detail & Related papers  (2021-03-24T03:14:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.