Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint

• URL: http://arxiv.org/abs/2410.15391v1
• Date: Sun, 20 Oct 2024 13:41:50 GMT
• Title: Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint
• Authors: Junwei Zhou, Xueting Li, Lu Qi, Ming-Hsuan Yang,
• Abstract summary: We present a framework that allows controllable and compositional 3D generation from text prompts. Our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation.
• Score: 61.25279122171029
• License:
• Abstract: We present Layout-Your-3D, a framework that allows controllable and compositional 3D generation from text prompts. Existing text-to-3D methods often struggle to generate assets with plausible object interactions or require tedious optimization processes. To address these challenges, our approach leverages 2D layouts as a blueprint to facilitate precise and plausible control over 3D generation. Starting with a 2D layout provided by a user or generated from a text description, we first create a coarse 3D scene using a carefully designed initialization process based on efficient reconstruction models. To enforce coherent global 3D layouts and enhance the quality of instance appearances, we propose a collision-aware layout optimization process followed by instance-wise refinement. Experimental results demonstrate that Layout-Your-3D yields more reasonable and visually appealing compositional 3D assets while significantly reducing the time required for each prompt. Additionally, Layout-Your-3D can be easily applicable to downstream tasks, such as 3D editing and object insertion. Our project page is available at:https://colezwhy.github.io/layoutyour3d/

Related papers

• Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches [50.51643519253066]
  3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes.
  arXiv  Detail & Related papers  (2024-08-08T16:27:37Z)
• iControl3D: An Interactive System for Controllable 3D Scene Generation [57.048647153684485]
  iControl3D is a novel interactive system that empowers users to generate and render customizable 3D scenes with precise control. We leverage 3D meshes as an intermediary proxy to iteratively merge individual 2D diffusion-generated images into a cohesive and unified 3D scene representation. Our neural rendering interface enables users to build a radiance field of their scene online and navigate the entire scene.
  arXiv  Detail & Related papers  (2024-08-03T06:35:09Z)
• SceneTeller: Language-to-3D Scene Generation [15.209079637302905]
  Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it. Our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices.
  arXiv  Detail & Related papers  (2024-07-30T10:45:28Z)
• Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts [76.73043724587679]
  We propose a dialogue-based 3D scene editing approach, termed CE3D. Hash-Atlas represents 3D scene views, which transfers the editing of 3D scenes onto 2D atlas images. Results demonstrate that CE3D effectively integrates multiple visual models to achieve diverse editing visual effects.
  arXiv  Detail & Related papers  (2024-07-09T13:24:42Z)
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting [52.150502668874495]
  We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. GALA3D is a user-friendly, end-to-end framework for state-of-the-art scene-level 3D content generation and controllable editing.
  arXiv  Detail & Related papers  (2024-02-11T13:40:08Z)
• Control3D: Towards Controllable Text-to-3D Generation [107.81136630589263]
  We present a text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D. A 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF. We exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene.
  arXiv  Detail & Related papers  (2023-11-09T15:50:32Z)
• Learning 3D Scene Priors with 2D Supervision [37.79852635415233]
  We propose a new method to learn 3D scene priors of layout and shape without requiring any 3D ground truth. Our method represents a 3D scene as a latent vector, from which we can progressively decode to a sequence of objects characterized by their class categories. Experiments on 3D-FRONT and ScanNet show that our method outperforms state of the art in single-view reconstruction.
  arXiv  Detail & Related papers  (2022-11-25T15:03:32Z)
• Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms [22.188206636953794]
  We propose the task of generating novel 3D floor plans from existing 3D rooms. One uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts.
  arXiv  Detail & Related papers  (2021-12-10T16:17:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.