GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

• URL: http://arxiv.org/abs/2104.07659v1
• Date: Thu, 15 Apr 2021 17:59:38 GMT
• Title: GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds
• Authors: Zekun Hao, Arun Mallya, Serge Belongie, Ming-Yu Liu
• Abstract summary: We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training.
• Score: 29.533111314655788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present GANcraft, an unsupervised neural rendering framework for generating photorealistic images of large 3D block worlds such as those created in Minecraft. Our method takes a semantic block world as input, where each block is assigned a semantic label such as dirt, grass, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images for a user-controlled camera. In the absence of paired ground truth real images for the block world, we devise a training technique based on pseudo-ground truth and adversarial training. This stands in contrast to prior work on neural rendering for view synthesis, which requires ground truth images to estimate scene geometry and view-dependent appearance. In addition to camera trajectory, GANcraft allows user control over both scene semantics and output style. Experimental results with comparison to strong baselines show the effectiveness of GANcraft on this novel task of photorealistic 3D block world synthesis. The project website is available at https://nvlabs.github.io/GANcraft/ .

Related papers

• SceneCraft: Layout-Guided 3D Scene Generation [29.713491313796084]
  SceneCraft is a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences. Our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality.
  arXiv  Detail & Related papers  (2024-10-11T17:59:58Z)
• Synthesizing Moving People with 3D Control [88.68284137105654]
  We present a diffusion model-based framework for animating people from a single image for a given target 3D motion sequence. For the first part, we learn an in-filling diffusion model to hallucinate unseen parts of a person given a single image. Second, we develop a diffusion-based rendering pipeline, which is controlled by 3D human poses.
  arXiv  Detail & Related papers  (2024-01-19T18:59:11Z)
• R2Human: Real-Time 3D Human Appearance Rendering from a Single Image [42.74145788079571]
  R2Human is the first approach for real-time inference and rendering of 3D human appearance from a single image. We present an end-to-end network that performs high-fidelity color reconstruction of visible areas and provides reliable color inference for occluded regions.
  arXiv  Detail & Related papers  (2023-12-10T08:59:43Z)
• Infinite Photorealistic Worlds using Procedural Generation [135.10236145573043]
  Infinigen is a procedural generator of photorealistic 3D scenes of the natural world. Every asset, from shape to texture, is generated from scratch via randomized mathematical rules.
  arXiv  Detail & Related papers  (2023-06-15T17:46:16Z)
• Persistent Nature: A Generative Model of Unbounded 3D Worlds [74.51149070418002]
  We present an extendable, planar scene layout grid that can be rendered from arbitrary camera poses via a 3D decoder and volume rendering. Based on this representation, we learn a generative world model solely from single-view internet photos. Our approach enables scene extrapolation beyond the fixed bounds of current 3D generative models, while also supporting a persistent, camera-independent world representation.
  arXiv  Detail & Related papers  (2023-03-23T17:59:40Z)
• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections [49.802462165826554]
  We present SceneDreamer, an unconditional generative model for unbounded 3D scenes. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations.
  arXiv  Detail & Related papers  (2023-02-02T18:59:16Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)
• SMPLpix: Neural Avatars from 3D Human Models [56.85115800735619]
  We bridge the gap between classic rendering and the latest generative networks operating in pixel space. We train a network that directly converts a sparse set of 3D mesh vertices into photorealistic images. We show the advantage over conventional differentiables both in terms of the level of photorealism and rendering efficiency.
  arXiv  Detail & Related papers  (2020-08-16T10:22:00Z)
• Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis with Rendering [0.0]
  We introduce a hybrid generative neural graphics pipeline for improving the visual fidelity of driving simulations. We form 2D semantic images from 3D scenery consisting of simple object models without textures. These semantic images are then converted into photorealistic RGB images with a state-of-the-art Generative Adrial Network (GAN) trained on real-world driving scenes.
  arXiv  Detail & Related papers  (2020-07-31T03:25:17Z)
• Learning Neural Light Transport [28.9247002210861]
  We present an approach for learning light transport in static and dynamic 3D scenes using a neural network. We find that our model is able to produce photorealistic renderings of static and dynamic scenes.
  arXiv  Detail & Related papers  (2020-06-05T13:26:05Z)
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images [38.952307525311625]
  We present BlockGAN, an image generative model that learns object-aware 3D scene representations directly from unlabelled 2D images. Inspired by the computer graphics pipeline, we design BlockGAN to learn to first generate 3D features of background and foreground objects, then combine them into 3D features for the wholes cene.
  arXiv  Detail & Related papers  (2020-02-20T19:41:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.