SceneCraft: Layout-Guided 3D Scene Generation

• URL: http://arxiv.org/abs/2410.09049v1
• Date: Fri, 11 Oct 2024 17:59:58 GMT
• Title: SceneCraft: Layout-Guided 3D Scene Generation
• Authors: Xiuyu Yang, Yunze Man, Jun-Kun Chen, Yu-Xiong Wang,
• Abstract summary: SceneCraft is a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences. Our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality.
• Score: 29.713491313796084
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The creation of complex 3D scenes tailored to user specifications has been a tedious and challenging task with traditional 3D modeling tools. Although some pioneering methods have achieved automatic text-to-3D generation, they are generally limited to small-scale scenes with restricted control over the shape and texture. We introduce SceneCraft, a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences provided by users. Central to our method is a rendering-based technique, which converts 3D semantic layouts into multi-view 2D proxy maps. Furthermore, we design a semantic and depth conditioned diffusion model to generate multi-view images, which are used to learn a neural radiance field (NeRF) as the final scene representation. Without the constraints of panorama image generation, we surpass previous methods in supporting complicated indoor space generation beyond a single room, even as complicated as a whole multi-bedroom apartment with irregular shapes and layouts. Through experimental analysis, we demonstrate that our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality. Code and more results are available at: https://orangesodahub.github.io/SceneCraft

Related papers

• Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches [50.51643519253066]
  3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes.
  arXiv  Detail & Related papers  (2024-08-08T16:27:37Z)
• iControl3D: An Interactive System for Controllable 3D Scene Generation [57.048647153684485]
  iControl3D is a novel interactive system that empowers users to generate and render customizable 3D scenes with precise control. We leverage 3D meshes as an intermediary proxy to iteratively merge individual 2D diffusion-generated images into a cohesive and unified 3D scene representation. Our neural rendering interface enables users to build a radiance field of their scene online and navigate the entire scene.
  arXiv  Detail & Related papers  (2024-08-03T06:35:09Z)
• Invisible Stitch: Generating Smooth 3D Scenes with Depth Inpainting [75.7154104065613]
  We introduce a novel depth completion model, trained via teacher distillation and self-training to learn the 3D fusion process. We also introduce a new benchmarking scheme for scene generation methods that is based on ground truth geometry.
  arXiv  Detail & Related papers  (2024-04-30T17:59:40Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• Disentangled 3D Scene Generation with Layout Learning [109.03233745767062]
  We introduce a method to generate 3D scenes that are disentangled into their component objects. Our key insight is that objects can be discovered by finding parts of a 3D scene that, when rearranged spatially, still produce valid configurations of the same scene. We show that despite its simplicity, our approach successfully generates 3D scenes into individual objects.
  arXiv  Detail & Related papers  (2024-02-26T18:54:15Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• Ctrl-Room: Controllable Text-to-3D Room Meshes Generation with Layout Constraints [35.073500525250346]
  We present Ctrl-Room, which can generate convincing 3D rooms with designer-style layouts and high-fidelity textures from just a text prompt. Ctrl-Room enables versatile interactive editing operations such as resizing or moving individual furniture items.
  arXiv  Detail & Related papers  (2023-10-05T15:29:52Z)
• Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields [29.907615852310204]
  We present Text2NeRF, which is able to generate a wide range of 3D scenes purely from a text prompt. Our method requires no additional training data but only a natural language description of the scene as the input.
  arXiv  Detail & Related papers  (2023-05-19T10:58:04Z)
• Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images [18.24156991697044]
  We present a method for creating 3D indoor scenes with a generative model learned from semantic-segmented depth images. Given a room with a specified size, our method automatically generates 3D objects in a room from a randomly sampled latent code. Compared to existing methods, our method not only efficiently reduces the workload of modeling and acquiring 3D scenes for training, but also produces better object shapes.
  arXiv  Detail & Related papers  (2021-08-20T06:22:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.