ControlRoom3D: Room Generation using Semantic Proxy Rooms
- URL: http://arxiv.org/abs/2312.05208v1
- Date: Fri, 8 Dec 2023 17:55:44 GMT
- Title: ControlRoom3D: Room Generation using Semantic Proxy Rooms
- Authors: Jonas Schult, Sam Tsai, Lukas H\"ollein, Bichen Wu, Jialiang Wang,
Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, Peizhao
Zhang, Bastian Leibe, Peter Vajda, Ji Hou
- Abstract summary: We present ControlRoom3D, a novel method to generate high-quality room meshes.
Our approach is a user-defined 3D semantic proxy room that outlines a rough room layout.
When rendered to 2D, this 3D representation provides valuable geometric and semantic information to control powerful 2D models.
- Score: 48.93419701713694
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Manually creating 3D environments for AR/VR applications is a complex process
requiring expert knowledge in 3D modeling software. Pioneering works facilitate
this process by generating room meshes conditioned on textual style
descriptions. Yet, many of these automatically generated 3D meshes do not
adhere to typical room layouts, compromising their plausibility, e.g., by
placing several beds in one bedroom. To address these challenges, we present
ControlRoom3D, a novel method to generate high-quality room meshes. Central to
our approach is a user-defined 3D semantic proxy room that outlines a rough
room layout based on semantic bounding boxes and a textual description of the
overall room style. Our key insight is that when rendered to 2D, this 3D
representation provides valuable geometric and semantic information to control
powerful 2D models to generate 3D consistent textures and geometry that aligns
well with the proxy room. Backed up by an extensive study including
quantitative metrics and qualitative user evaluations, our method generates
diverse and globally plausible 3D room meshes, thus empowering users to design
3D rooms effortlessly without specialized knowledge.
Related papers
- SceneCraft: Layout-Guided 3D Scene Generation [29.713491313796084]
SceneCraft is a novel method for generating detailed indoor scenes that adhere to textual descriptions and spatial layout preferences.
Our method significantly outperforms existing approaches in complex indoor scene generation with diverse textures, consistent geometry, and realistic visual quality.
arXiv Detail & Related papers (2024-10-11T17:59:58Z) - SceneTeller: Language-to-3D Scene Generation [15.209079637302905]
Given a prompt in natural language describing the object placement in the room, our method produces a high-quality 3D scene corresponding to it.
Our turnkey pipeline produces state-of-the-art 3D scenes, while being easy to use even for novices.
arXiv Detail & Related papers (2024-07-30T10:45:28Z) - Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D
Prior [52.44678180286886]
2D diffusion models find a distillation approach that achieves excellent generalization and rich details without any 3D data.
We propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity, generalizability, and geometric consistency simultaneously.
arXiv Detail & Related papers (2023-12-11T18:59:18Z) - IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts [90.49024750432139]
We present IPDreamer, a novel method that captures intricate appearance features from complex $textbfI$mage $textbfP$rompts and aligns the synthesized 3D object with these extracted features.
Our experiments demonstrate that IPDreamer consistently generates high-quality 3D objects that align with both the textual and complex image prompts.
arXiv Detail & Related papers (2023-10-09T03:11:08Z) - Estimating Generic 3D Room Structures from 2D Annotations [36.2713028459562]
We propose a novel method to produce generic 3D room layouts just from 2D segmentation masks.
Based on these 2D annotations, we automatically reconstruct 3D plane equations for the structural elements and their spatial extent in the scene.
We release 2246 3D room layouts on the RealEstate10k dataset, containing YouTube videos.
arXiv Detail & Related papers (2023-06-15T12:10:27Z) - CC3D: Layout-Conditioned Generation of Compositional 3D Scenes [49.281006972028194]
We introduce CC3D, a conditional generative model that synthesizes complex 3D scenes conditioned on 2D semantic scene layouts.
Our evaluations on synthetic 3D-FRONT and real-world KITTI-360 datasets demonstrate that our model generates scenes of improved visual and geometric quality.
arXiv Detail & Related papers (2023-03-21T17:59:02Z) - SceneHGN: Hierarchical Graph Networks for 3D Indoor Scene Generation
with Fine-Grained Geometry [92.24144643757963]
3D indoor scenes are widely used in computer graphics, with applications ranging from interior design to gaming to virtual and augmented reality.
High-quality 3D indoor scenes are highly demanded while it requires expertise and is time-consuming to design high-quality 3D indoor scenes manually.
We propose SCENEHGN, a hierarchical graph network for 3D indoor scenes that takes into account the full hierarchy from the room level to the object level, then finally to the object part level.
For the first time, our method is able to directly generate plausible 3D room content, including furniture objects with fine-grained geometry, and
arXiv Detail & Related papers (2023-02-16T15:31:59Z) - 3D-Aware Indoor Scene Synthesis with Depth Priors [62.82867334012399]
Existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside.
We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry.
arXiv Detail & Related papers (2022-02-17T09:54:29Z) - Roominoes: Generating Novel 3D Floor Plans From Existing 3D Rooms [22.188206636953794]
We propose the task of generating novel 3D floor plans from existing 3D rooms.
One uses available 2D floor plans to guide selection and deformation of 3D rooms; the other learns to retrieve a set of compatible 3D rooms and combine them into novel layouts.
arXiv Detail & Related papers (2021-12-10T16:17:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.