Unconstrained Scene Generation with Locally Conditioned Radiance Fields

• URL: http://arxiv.org/abs/2104.00670v1
• Date: Thu, 1 Apr 2021 17:58:26 GMT
• Title: Unconstrained Scene Generation with Locally Conditioned Radiance Fields
• Authors: Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, Joshua M. Susskind
• Abstract summary: We introduce Generative Scene Networks (GSN), which learns to decompose scenes into a collection of local radiance fields. Our model can be used as a prior to generate new scenes, or to complete a scene given only sparse 2D observations.
• Score: 24.036609880683585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We tackle the challenge of learning a distribution over complex, realistic, indoor scenes. In this paper, we introduce Generative Scene Networks (GSN), which learns to decompose scenes into a collection of many local radiance fields that can be rendered from a free moving camera. Our model can be used as a prior to generate new scenes, or to complete a scene given only sparse 2D observations. Recent work has shown that generative models of radiance fields can capture properties such as multi-view consistency and view-dependent lighting. However, these models are specialized for constrained viewing of single objects, such as cars or faces. Due to the size and complexity of realistic indoor environments, existing models lack the representational capacity to adequately capture them. Our decomposition scheme scales to larger and more complex scenes while preserving details and diversity, and the learned prior enables high-quality rendering from viewpoints that are significantly different from observed viewpoints. When compared to existing models, GSN produces quantitatively higher-quality scene renderings across several different scene datasets.

Related papers

• Content-Aware Radiance Fields: Aligning Model Complexity with Scene Intricacy Through Learned Bitwidth Quantization [30.707317084452594]
  radiance field models are designed to represent 3D content by that training models for each individual scene. This unique characteristic of scene representation and per-scene training distinguishes radiance field models from other neural models. We propose content?aware radiance fields, aligning the model complexity with the scene intricacies through Adversarial Content-Aware Quantization (A-CAQ)
  arXiv  Detail & Related papers  (2024-10-25T11:39:55Z)
• DORSal: Diffusion for Object-centric Representations of Scenes et al [28.181157214966493]
  Recent progress in 3D scene understanding enables scalable learning of representations across large datasets of diverse scenes. We propose DORSal, which adapts a video diffusion architecture for 3D scene generation conditioned on frozen object-centric slot-based representations of scenes.
  arXiv  Detail & Related papers  (2023-06-13T18:32:35Z)
• NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models [85.20004959780132]
  We introduce NeuralField-LDM, a generative model capable of synthesizing complex 3D environments. We show how NeuralField-LDM can be used for a variety of 3D content creation applications, including conditional scene generation, scene inpainting and scene style manipulation.
  arXiv  Detail & Related papers  (2023-04-19T16:13:21Z)
• GAUDI: A Neural Architect for Immersive 3D Scene Generation [67.97817314857917]
  GAUDI is a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera. We show that GAUDI obtains state-of-the-art performance in the unconditional generative setting across multiple datasets.
  arXiv  Detail & Related papers  (2022-07-27T19:10:32Z)
• Light Field Neural Rendering [47.7586443731997]
  Methods based on geometric reconstruction need only sparse views, but cannot accurately model non-Lambertian effects. We introduce a model that combines the strengths and mitigates the limitations of these two directions. Our model outperforms the state-of-the-art on multiple forward-facing and 360deg datasets.
  arXiv  Detail & Related papers  (2021-12-17T18:58:05Z)
• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering [145.95688637309746]
  We introduce BungeeNeRF, a progressive neural radiance field that achieves level-of-detail rendering across drastically varied scales. We demonstrate the superiority of BungeeNeRF in modeling diverse multi-scale scenes with drastically varying views on multiple data sources.
  arXiv  Detail & Related papers  (2021-12-10T13:16:21Z)
• RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs [79.00855490550367]
  We show that NeRF can produce photorealistic renderings of unseen viewpoints when many input views are available. We address this by regularizing the geometry and appearance of patches rendered from unobserved viewpoints. Our model outperforms not only other methods that optimize over a single scene, but also conditional models that are extensively pre-trained on large multi-view datasets.
  arXiv  Detail & Related papers  (2021-12-01T18:59:46Z)
• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields [45.21191307444531]
  Deep generative models allow for photorealistic image synthesis at high resolutions. But for many applications, this is not enough: content creation also needs to be controllable. Our key hypothesis is that incorporating a compositional 3D scene representation into the generative model leads to more controllable image synthesis.
  arXiv  Detail & Related papers  (2020-11-24T14:14:15Z)
• Neural Reflectance Fields for Appearance Acquisition [61.542001266380375]
  We present Neural Reflectance Fields, a novel deep scene representation that encodes volume density, normal and reflectance properties at any 3D point in a scene. We combine this representation with a physically-based differentiable ray marching framework that can render images from a neural reflectance field under any viewpoint and light.
  arXiv  Detail & Related papers  (2020-08-09T22:04:36Z)
• Neural Sparse Voxel Fields [151.20366604586403]
  We introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results.
  arXiv  Detail & Related papers  (2020-07-22T17:51:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.