EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion

• URL: http://arxiv.org/abs/2405.00915v1
• Date: Thu, 2 May 2024 00:04:02 GMT
• Title: EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusion
• Authors: Guangyao Zhai, Evin Pınar Örnek, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, Benjamin Busam,
• Abstract summary: We present EchoScene, an interactive and controllable generative model that generates 3D indoor scenes on scene graphs. Existing methods struggle to handle scene graphs due to varying numbers of nodes, multiple edge combinations, and manipulator-induced node-edge operations.
• Score: 77.0556470600979
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present EchoScene, an interactive and controllable generative model that generates 3D indoor scenes on scene graphs. EchoScene leverages a dual-branch diffusion model that dynamically adapts to scene graphs. Existing methods struggle to handle scene graphs due to varying numbers of nodes, multiple edge combinations, and manipulator-induced node-edge operations. EchoScene overcomes this by associating each node with a denoising process and enables collaborative information exchange, enhancing controllable and consistent generation aware of global constraints. This is achieved through an information echo scheme in both shape and layout branches. At every denoising step, all processes share their denoising data with an information exchange unit that combines these updates using graph convolution. The scheme ensures that the denoising processes are influenced by a holistic understanding of the scene graph, facilitating the generation of globally coherent scenes. The resulting scenes can be manipulated during inference by editing the input scene graph and sampling the noise in the diffusion model. Extensive experiments validate our approach, which maintains scene controllability and surpasses previous methods in generation fidelity. Moreover, the generated scenes are of high quality and thus directly compatible with off-the-shelf texture generation. Code and trained models are open-sourced.

Related papers

• Generating Human Interaction Motions in Scenes with Text Control [66.74298145999909]
  We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model. To facilitate training, we embed annotated navigation and interaction motions within scenes.
  arXiv  Detail & Related papers  (2024-04-16T16:04:38Z)
• InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior [27.773451301040424]
  InstructScene is a novel generative framework that integrates a semantic graph prior and a layout decoder. We show that the proposed method surpasses existing state-of-the-art approaches by a large margin.
  arXiv  Detail & Related papers  (2024-02-07T10:09:00Z)
• Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation [51.92419880088668]
  We propose a novel DynSGG model based on multi-task learning, DynSGG-MTL, which introduces the local interaction information and global human-action interaction information. Long-temporal human actions supervise the model to generate multiple scene graphs that conform to the global constraints and avoid the model being unable to learn the tail predicates.
  arXiv  Detail & Related papers  (2023-08-10T01:24:25Z)
• 3D Scene Diffusion Guidance using Scene Graphs [3.207455883863626]
  We propose a novel approach for 3D scene diffusion guidance using scene graphs. To leverage the relative spatial information the scene graphs provide, we make use of relational graph convolutional blocks within our denoising network.
  arXiv  Detail & Related papers  (2023-08-08T06:16:37Z)
• Unsupervised Coordinate-Based Video Denoising [2.867801048665443]
  We introduce a novel unsupervised video denoising deep learning approach that can help to mitigate data scarcity issues. Our method comprises three modules: a Feature generator creating features maps, a Denoise-Net generating denoised but slightly blurry reference frames, and a Refine-Net re-introducing high-frequency details.
  arXiv  Detail & Related papers  (2023-07-01T00:11:40Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training [112.94542676251133]
  We propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections.
  arXiv  Detail & Related papers  (2022-11-21T01:11:19Z)
• Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes Using Scene Graphs [85.54212143154986]
  Controllable scene synthesis consists of generating 3D information that satisfy underlying specifications. Scene graphs are representations of a scene composed of objects (nodes) and inter-object relationships (edges) We propose the first work that directly generates shapes from a scene graph in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-08-19T17:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.