iControl3D: An Interactive System for Controllable 3D Scene Generation

• URL: http://arxiv.org/abs/2408.01678v1
• Date: Sat, 3 Aug 2024 06:35:09 GMT
• Title: iControl3D: An Interactive System for Controllable 3D Scene Generation
• Authors: Xingyi Li, Yizheng Wu, Jun Cen, Juewen Peng, Kewei Wang, Ke Xian, Zhe Wang, Zhiguo Cao, Guosheng Lin,
• Abstract summary: iControl3D is a novel interactive system that empowers users to generate and render customizable 3D scenes with precise control. We leverage 3D meshes as an intermediary proxy to iteratively merge individual 2D diffusion-generated images into a cohesive and unified 3D scene representation. Our neural rendering interface enables users to build a radiance field of their scene online and navigate the entire scene.
• Score: 57.048647153684485
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D content creation has long been a complex and time-consuming process, often requiring specialized skills and resources. While recent advancements have allowed for text-guided 3D object and scene generation, they still fall short of providing sufficient control over the generation process, leading to a gap between the user's creative vision and the generated results. In this paper, we present iControl3D, a novel interactive system that empowers users to generate and render customizable 3D scenes with precise control. To this end, a 3D creator interface has been developed to provide users with fine-grained control over the creation process. Technically, we leverage 3D meshes as an intermediary proxy to iteratively merge individual 2D diffusion-generated images into a cohesive and unified 3D scene representation. To ensure seamless integration of 3D meshes, we propose to perform boundary-aware depth alignment before fusing the newly generated mesh with the existing one in 3D space. Additionally, to effectively manage depth discrepancies between remote content and foreground, we propose to model remote content separately with an environment map instead of 3D meshes. Finally, our neural rendering interface enables users to build a radiance field of their scene online and navigate the entire scene. Extensive experiments have been conducted to demonstrate the effectiveness of our system. The code will be made available at https://github.com/xingyi-li/iControl3D.

Related papers

• SceneCraft: Layout-Guided 3D Scene Generation [29.713491313796084]
  SceneCraft is a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences. Our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality.
  arXiv  Detail & Related papers  (2024-10-11T17:59:58Z)
• Chat-Edit-3D: Interactive 3D Scene Editing via Text Prompts [76.73043724587679]
  We propose a dialogue-based 3D scene editing approach, termed CE3D. Hash-Atlas represents 3D scene views, which transfers the editing of 3D scenes onto 2D atlas images. Results demonstrate that CE3D effectively integrates multiple visual models to achieve diverse editing visual effects.
  arXiv  Detail & Related papers  (2024-07-09T13:24:42Z)
• Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning [52.81032340916171]
  Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes. Our method achieves superior controllability and flexibility in the 3D assets generation task.
  arXiv  Detail & Related papers  (2024-05-13T17:56:13Z)
• Interactive3D: Create What You Want by Interactive 3D Generation [13.003964182554572]
  We introduce Interactive3D, an innovative framework for interactive 3D generation that grants users precise control over the generative process. Our experiments demonstrate that Interactive3D markedly improves the controllability and quality of 3D generation.
  arXiv  Detail & Related papers  (2024-04-25T11:06:57Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting [52.150502668874495]
  We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. GALA3D is a user-friendly, end-to-end framework for state-of-the-art scene-level 3D content generation and controllable editing.
  arXiv  Detail & Related papers  (2024-02-11T13:40:08Z)
• SceneWiz3D: Towards Text-guided 3D Scene Composition [134.71933134180782]
  Existing approaches either leverage large text-to-image models to optimize a 3D representation or train 3D generators on object-centric datasets. We introduce SceneWiz3D, a novel approach to synthesize high-fidelity 3D scenes from text.
  arXiv  Detail & Related papers  (2023-12-13T18:59:30Z)
• LucidDreaming: Controllable Object-Centric 3D Generation [10.646855651524387]
  We present a pipeline capable of spatial and numerical control over 3D generation from only textual prompt commands or 3D bounding boxes. LucidDreaming achieves superior results in object placement precision and generation fidelity compared to current approaches.
  arXiv  Detail & Related papers  (2023-11-30T18:55:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.