DVSM: Decoder-only View Synthesis Model Done Right

• URL: http://arxiv.org/abs/2605.29891v1
• Date: Thu, 28 May 2026 13:16:09 GMT
• Title: DVSM: Decoder-only View Synthesis Model Done Right
• Abstract summary: Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks.<n>We show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity.<n>Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.
• Score: 53.493403264581424
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent Large View Synthesis Models (LVSMs) advocate an encoder-decoder architecture that separates reconstruction and rendering into distinct networks. We re-examine this design. Through controlled experiments, we show that a decoder-only architecture, which represents scenes implicitly as a KV-cache, outperforms encoder-decoder variants while using fewer parameters at identical rendering complexity. Further analysis shows that sharing weights between the color-input reconstruction network and the camera-only rendering network better aligns their features at the same viewpoint, facilitating image synthesis. Building on this finding, our model, dubbed DVSM, further incorporates foundation model priors and stage-wise patch sizing for an improved efficiency-quality tradeoff. Our results establish a new state of the art for novel-view synthesis across multiple benchmarks, in some cases even outperforming per-scene-optimized 3DGS under dense input views.

Related papers

• Scaling View Synthesis Transformers [14.587384620293784]
  Geometry-free view synthesis transformers have recently achieved state-of-the-art performance in Novel View Synthesis (NVS)<n>We present a systematic study of scaling laws for view synthesis transformers and derive design principles for training compute-optimal NVS models.
  arXiv  Detail & Related papers  (2026-02-24T20:18:55Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• DeCo-VAE: Learning Compact Latents for Video Reconstruction via Decoupled Representation [14.242798717551471]
  We propose decoupled VAE (Co-VAE) to achieve compact latent representation.<n>We design dedicated encoders for each decoupled component and adopt a shared 3D decoder to maintain consistency during reconstruction.
  arXiv  Detail & Related papers  (2025-11-18T14:34:20Z)
• MsEdF: A Multi-stream Encoder-decoder Framework for Remote Sensing Image Captioning [2.435006380732194]
  Remote sensing images contain complex spatial patterns and semantic structures, which makes the captioning model difficult to accurately describe.<n>We propose a novel Multi-stream-decoder Framework (MsEdF) which improves the performance of RSIC by optimizing both the spatial representation and language generation.<n>Experiments on three benchmark RSIC datasets show that MsEdF outperforms several baseline models.
  arXiv  Detail & Related papers  (2025-02-13T12:54:13Z)
• LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias [50.13457154615262]
  We propose a transformer-based approach for scalable and generalizable novel view synthesis from sparse-view inputs.<n>We introduce two architectures: (1) an encoder-decoder LVSM, which encodes input image tokens into a fixed number of 1D latent tokens; and (2) a decoder-only LVSM, which directly maps input images to novel-view outputs.
  arXiv  Detail & Related papers  (2024-10-22T17:58:28Z)
• VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision [59.632286735304156]
  It is more efficient to enhance/analyze the coded representations directly without decoding them into pixels. We propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis.
  arXiv  Detail & Related papers  (2023-06-19T03:04:57Z)
• DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decoder [73.1010640692609]
  We propose a VQ-VAE architecture model with a diffusion decoder (DiVAE) to work as the reconstructing component in image synthesis. Our model achieves state-of-the-art results and generates more photorealistic images specifically.
  arXiv  Detail & Related papers  (2022-06-01T10:39:12Z)
• GAN-Based Multi-View Video Coding with Spatio-Temporal EPI Reconstruction [19.919826392704472]
  We propose a novel multi-view video coding method that leverages the image generation capabilities of Generative Adrial Network (GAN) At the encoder, we construct atemporal Epipolar Plane Image (EPI) decoder and further utilize a convolutional network to extract the latent code of a GAN as Side Information (SI) At the side, we combine SI and adjacent viewpoints to reconstruct intermediate views using the GAN generator.
  arXiv  Detail & Related papers  (2022-05-07T08:52:54Z)
• An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal [99.49099501559652]
  Video Coding for Machine (VCM) aims to bridge the gap between visual feature compression and classical video coding. We employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern. By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames.
  arXiv  Detail & Related papers  (2020-01-09T14:18:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.