Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction

• URL: http://arxiv.org/abs/2311.18695v1
• Date: Thu, 30 Nov 2023 16:42:24 GMT
• Title: Seg2Reg: Differentiable 2D Segmentation to 1D Regression Rendering for 360 Room Layout Reconstruction
• Authors: Cheng Sun, Wei-En Tai, Yu-Lin Shih, Kuan-Wei Chen, Yong-Jing Syu, Kent Selwyn The, Yu-Chiang Frank Wang, Hwann-Tzong Chen
• Abstract summary: We present Seg2Reg to render 1D layout depth regression from the 2D segmentation map. Specifically, our model predicts floor-plan density for the input equirectangular 360-degree image. We propose a novel 3D warping augmentation on layout to improve generalization.
• Score: 31.58900366831744
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: State-of-the-art single-view 360-degree room layout reconstruction methods formulate the problem as a high-level 1D (per-column) regression task. On the other hand, traditional low-level 2D layout segmentation is simpler to learn and can represent occluded regions, but it requires complex post-processing for the targeting layout polygon and sacrifices accuracy. We present Seg2Reg to render 1D layout depth regression from the 2D segmentation map in a differentiable and occlusion-aware way, marrying the merits of both sides. Specifically, our model predicts floor-plan density for the input equirectangular 360-degree image. Formulating the 2D layout representation as a density field enables us to employ `flattened' volume rendering to form 1D layout depth regression. In addition, we propose a novel 3D warping augmentation on layout to improve generalization. Finally, we re-implement recent room layout reconstruction methods into our codebase for benchmarking and explore modern backbones and training techniques to serve as the strong baseline. Our model significantly outperforms previous arts. The code will be made available upon publication.

Related papers

• Sp2360: Sparse-view 360 Scene Reconstruction using Cascaded 2D Diffusion Priors [51.36238367193988]
  We tackle sparse-view reconstruction of a 360 3D scene using priors from latent diffusion models (LDM) We present SparseSplat360, a method that employs a cascade of in-painting and artifact removal models to fill in missing details and clean novel views. Our method generates entire 360 scenes from as few as 9 input views, with a high degree of foreground and background detail.
  arXiv  Detail & Related papers  (2024-05-26T11:01:39Z)
• Self-Supervised Geometry-Aware Encoder for Style-Based 3D GAN Inversion [115.82306502822412]
  StyleGAN has achieved great progress in 2D face reconstruction and semantic editing via image inversion and latent editing. A corresponding generic 3D GAN inversion framework is still missing, limiting the applications of 3D face reconstruction and semantic editing. We study the challenging problem of 3D GAN inversion where a latent code is predicted given a single face image to faithfully recover its 3D shapes and detailed textures.
  arXiv  Detail & Related papers  (2022-12-14T18:49:50Z)
• SimpleRecon: 3D Reconstruction Without 3D Convolutions [21.952478592241]
  We show how focusing on high quality multi-view depth prediction leads to highly accurate 3D reconstructions using simple off-the-shelf depth fusion. Our method achieves a significant lead over the current state-of-the-art for depth estimation and close or better for 3D reconstruction on ScanNet and 7-Scenes.
  arXiv  Detail & Related papers  (2022-08-31T09:46:34Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• Neural 3D Scene Reconstruction with the Manhattan-world Assumption [58.90559966227361]
  This paper addresses the challenge of reconstructing 3D indoor scenes from multi-view images. Planar constraints can be conveniently integrated into the recent implicit neural representation-based reconstruction methods. The proposed method outperforms previous methods by a large margin on 3D reconstruction quality.
  arXiv  Detail & Related papers  (2022-05-05T17:59:55Z)
• 3D-Aware Indoor Scene Synthesis with Depth Priors [62.82867334012399]
  Existing methods fail to model indoor scenes due to the large diversity of room layouts and the objects inside. We argue that indoor scenes do not have a shared intrinsic structure, and hence only using 2D images cannot adequately guide the model with the 3D geometry.
  arXiv  Detail & Related papers  (2022-02-17T09:54:29Z)
• LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Rendering [59.63979143021241]
  We formulate the task of 360 layout estimation as a problem of predicting depth on the horizon line of a panorama. We propose the Differentiable Depth Rendering procedure to make the conversion from layout to depth prediction differentiable. Our method achieves state-of-the-art performance on numerous 360 layout benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T15:48:41Z)
• General 3D Room Layout from a Single View by Render-and-Compare [36.94817376590415]
  We present a novel method to reconstruct the 3D layout of a room from a single perspective view. Our dataset consists of 293 images from ScanNet, which we annotated with precise 3D layouts.
  arXiv  Detail & Related papers  (2020-01-07T16:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.