RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video

• URL: http://arxiv.org/abs/2605.31535v1
• Date: Fri, 29 May 2026 16:50:27 GMT
• Title: RayDer: Scalable Self-Supervised Novel View Synthesis from Real-World Video
• Abstract summary: RayDer is a feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone.<n>A minimal dynamic state, treated as a nuisance factor, absorbs time-varying content and enables stable training on real-world video.<n>RayDer exhibits clean power-law scaling with data and compute, and outperforms static-scene data mixtures.
• Score: 24.21520175195074
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised novel view synthesis (NVS) remains challenging to scale, despite the abundance of video data, largely due to the brittleness of training on realistic videos and the hard-to-predict scaling behavior of multi-network system designs. We introduce RayDer, a unified, feed-forward transformer that consolidates camera estimation, scene reconstruction, and rendering into a single backbone, turning self-supervised NVS into a well-posed single-model scaling problem. A minimal dynamic state, treated as a nuisance factor, absorbs time-varying content and enables stable training on unconstrained real-world video. Importantly, RayDer keeps static-scene NVS as its target task: dynamic content is leveraged purely as scalable supervision, not reconstructed as in dynamic-scene (4D) NVS. Across multiple model sizes and orders of magnitude in data, RayDer exhibits clean power-law scaling with data and compute, and outperforms static-scene data mixtures. On a large number of benchmarks, RayDer achieves strong zero-shot open-set performance competitive with state-of-the-art supervised approaches. Project Page: https://compvis.github.io/rayder

Related papers

• Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement [2.931556701508956]
  We introduce NOVA, a world modelling framework that represents the system state as the weights and biases of an auxiliary coordinate-based implicit neural representation (INR)<n>This structured representation is analytically rendered, which eliminates the decoder bottleneck while conferring compactness, portability, and zero-shot super-resolution.<n>We validate our framework on several challenging datasets, achieving strong controllable forecasting while operating on a single consumer GPU at $sim$40M parameters.
  arXiv  Detail & Related papers  (2026-05-07T14:02:00Z)
• FreeScale: Scaling 3D Scenes via Certainty-Aware Free-View Generation [75.74617373156902]
  FreeScale is a framework that transforms limited real-world image sequences into a scalable source of high-quality training data.<n>We show FreeScale's effectiveness by scaling up the training of feedforward NVS models, achieving a notable gain of 2.7 dB in PSNR.<n>Our work provides a practical and powerful data generation engine to overcome a fundamental bottleneck in 3D vision.
  arXiv  Detail & Related papers  (2026-04-12T08:00:53Z)
• WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments [12.449699948290215]
  We present WildRayZer, a framework for novel view synthesis (NVS) in dynamic environments where both the camera and objects move.<n>Experiments show that WildRayZer consistently outperforms optimization-based and feed-forward baselines in both transient-region removal and full-frame NVS quality.
  arXiv  Detail & Related papers  (2026-01-15T18:59:58Z)
• Feed-Forward Bullet-Time Reconstruction of Dynamic Scenes from Monocular Videos [110.3924779333809]
  We present BTimer, the first motion-aware feed-forward model for real-time reconstruction and novel view synthesis of dynamic scenes.<n>Our approach reconstructs the full scene in a 3D Gaussian Splatting representation at a given target ('bullet') timestamp by aggregating information from all the context frames.<n>Given a casual monocular dynamic video, BTimer reconstructs a bullet-time scene within 150ms while reaching state-of-the-art performance on both static and dynamic scene datasets.
  arXiv  Detail & Related papers  (2024-12-04T18:15:06Z)
• MegaScenes: Scene-Level View Synthesis at Scale [69.21293001231993]
  Scene-level novel view synthesis (NVS) is fundamental to many vision and graphics applications. We create a large-scale scene-level dataset from Internet photo collections, called MegaScenes, which contains over 100K structure from motion (SfM) reconstructions from around the world. We analyze failure cases of state-of-the-art NVS methods and significantly improve generation consistency.
  arXiv  Detail & Related papers  (2024-06-17T17:55:55Z)
• D-NPC: Dynamic Neural Point Clouds for Non-Rigid View Synthesis from Monocular Video [53.83936023443193]
  This paper contributes to the field by introducing a new synthesis method for dynamic novel view from monocular video, such as smartphone captures.<n>Our approach represents the as a $textitdynamic neural point cloud$, an implicit time-conditioned point cloud that encodes local geometry and appearance in separate hash-encoded neural feature grids.
  arXiv  Detail & Related papers  (2024-06-14T14:35:44Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.