Scaling View Synthesis Transformers

• URL: http://arxiv.org/abs/2602.21341v1
• Date: Tue, 24 Feb 2026 20:18:55 GMT
• Title: Scaling View Synthesis Transformers
• Abstract summary: Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS)<n>We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models.
• Score: 14.587384620293784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS), outperforming traditional approaches that rely on explicit geometry modeling. Yet the factors governing their scaling with compute remain unclear. We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models. Contrary to prior findings, we show that encoder-decoder architectures can be compute-optimal; we trace earlier negative results to suboptimal architectural choices and comparisons across unequal training compute budgets. Across several compute levels, we demonstrate that our encoder-decoder architecture, which we call the Scalable View Synthesis Model (SVSM), scales as effectively as decoder-only models, achieves a superior performance-compute Pareto frontier, and surpasses the previous state-of-the-art on real-world NVS benchmarks with substantially reduced training compute.

Related papers

• Compact Neural Appearance Models for Efficient Gaussian Splatting [1.579851447355843]
  primitive-based radiance fields such as 3D Gaussian Splatting typically model view-dependent appearance using low-order spherical harmonics (SH)<n>We present a thorough, end-to-end comparison of SH and recent spherical appearance models.<n>We introduce an implicit alternative that decodes compact per-primitive latent codes using a tiny shared pipeline.
  arXiv  Detail & Related papers  (2026-09-04T15:18:54Z)
• LevelSyn: Physical-Aware Logic Synthesis via Level-Asynchronous Graph Neural Networks [12.485442944403445]
  LevelSyn is a novel logic synthesis framework that integrates hierarchical representation learning with a wirelength-driven optimization engine.<n>At its core, LevelSyn leverages a level-asynchronous Graph Neural Network (GNN) to predict high-fidelity gate coordinates.<n>LevelSyn significantly outperforms state-of-the-art (SOTA) methods, achieving an average power reduction of 6.89% and a timing delay improvement of 27.48%.
  arXiv  Detail & Related papers  (2026-09-03T09:46:02Z)
• DVSM: Decoder-only View Synthesis Model Done Right [53.493403264581424]
  Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks.<n>We show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity.<n>Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.
  arXiv  Detail & Related papers  (2026-05-28T13:16:09Z)
• Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data [53.040873127309766]
  We propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning.<n>Our method outperforms existing models on both in-dataset and cross-dataset evaluations.
  arXiv  Detail & Related papers  (2025-09-08T17:58:06Z)
• SimpleGVR: A Simple Baseline for Latent-Cascaded Video Super-Resolution [46.311223206965934]
  We study key design principles for latter cascaded video super-resolution models, which are underexplored currently.<n>First, we propose two strategies to generate training pairs that better mimic the output characteristics of the base model, ensuring alignment between the VSR model and its upstream generator.<n>Second, we provide critical insights into VSR model behavior through systematic analysis of (1) timestep sampling strategies, (2) noise augmentation effects on low-resolution (LR) inputs.
  arXiv  Detail & Related papers  (2025-06-24T17:57:26Z)
• STAR: Synthesis of Tailored Architectures [61.080157488857516]
  We propose a new approach for the synthesis of tailored architectures (STAR)<n>Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics.<n>Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.
  arXiv  Detail & Related papers  (2024-11-26T18:42:42Z)
• Novel View Synthesis with Pixel-Space Diffusion Models [4.844800099745365]
  generative models are being increasingly employed in novel view synthesis (NVS) We adapt a modern diffusion model architecture for end-to-end NVS in the pixel space. We introduce a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance.
  arXiv  Detail & Related papers  (2024-11-12T12:58:33Z)
• Learning to Compare Hardware Designs for High-Level Synthesis [44.408523725466374]
  High-level synthesis (HLS) is an automated design process that transforms high-level code into hardware designs.<n>HLS relies on pragmas, which are directives inserted into the source code to guide the synthesis process.<n>We propose compareXplore, a novel approach that learns to compare hardware designs for effective HLS optimization.
  arXiv  Detail & Related papers  (2024-09-20T00:47:29Z)
• Mechanistic Design and Scaling of Hybrid Architectures [114.3129802943915]
  We identify and test new hybrid architectures constructed from a variety of computational primitives. We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis. We find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures.
  arXiv  Detail & Related papers  (2024-03-26T16:33:12Z)
• Uncovering mesa-optimization algorithms in Transformers [61.06055590704677]
  Some autoregressive models can learn as an input sequence is processed, without undergoing any parameter changes, and without being explicitly trained to do so. We show that standard next-token prediction error minimization gives rise to a subsidiary learning algorithm that adjusts the model as new inputs are revealed. Our findings explain in-context learning as a product of autoregressive loss minimization and inform the design of new optimization-based Transformer layers.
  arXiv  Detail & Related papers  (2023-09-11T22:42:50Z)
• Joint Hierarchical Priors and Adaptive Spatial Resolution for Efficient Neural Image Compression [11.25130799452367]
  We propose an absolute image compression transformer (ICT) for neural image compression (NIC) ICT captures both global and local contexts from the latent representations and better parameterize the distribution of the quantized latents. Our framework significantly improves the trade-off between coding efficiency and decoder complexity over the versatile video coding (VVC) reference encoder (VTM-18.0) and the neural SwinT-ChARM.
  arXiv  Detail & Related papers  (2023-07-05T13:17:14Z)
• Towards Theoretically Inspired Neural Initialization Optimization [66.04735385415427]
  We propose a differentiable quantity, named GradCosine, with theoretical insights to evaluate the initial state of a neural network. We show that both the training and test performance of a network can be improved by maximizing GradCosine under norm constraint. Generalized from the sample-wise analysis into the real batch setting, NIO is able to automatically look for a better initialization with negligible cost.
  arXiv  Detail & Related papers  (2022-10-12T06:49:16Z)
• Effective Invertible Arbitrary Image Rescaling [77.46732646918936]
  Invertible Neural Networks (INN) are able to increase upscaling accuracy significantly by optimizing the downscaling and upscaling cycle jointly. A simple and effective invertible arbitrary rescaling network (IARN) is proposed to achieve arbitrary image rescaling by training only one model in this work. It is shown to achieve a state-of-the-art (SOTA) performance in bidirectional arbitrary rescaling without compromising perceptual quality in LR outputs.
  arXiv  Detail & Related papers  (2022-09-26T22:22:30Z)
• A Graph Deep Learning Framework for High-Level Synthesis Design Space Exploration [11.154086943903696]
  High-Level Synthesis is a solution for fast prototyping application-specific hardware. We propose HLS, for the first time in the literature, graph neural networks that jointly predict acceleration performance and hardware costs. We show that our approach achieves prediction accuracy comparable with that of commonly used simulators.
  arXiv  Detail & Related papers  (2021-11-29T18:17:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.