MonoPlane: Exploiting Monocular Geometric Cues for Generalizable 3D Plane Reconstruction

• URL: http://arxiv.org/abs/2411.01226v1
• Date: Sat, 02 Nov 2024 12:15:29 GMT
• Title: MonoPlane: Exploiting Monocular Geometric Cues for Generalizable 3D Plane Reconstruction
• Authors: Wang Zhao, Jiachen Liu, Sheng Zhang, Yishu Li, Sili Chen, Sharon X Huang, Yong-Jin Liu, Hengkai Guo,
• Abstract summary: This paper presents a generalizable 3D plane detection and reconstruction framework named MonoPlane. We first leverage large-scale pre-trained neural networks to obtain the depth and surface normals from a single image. These monocular geometric cues are then incorporated into a proximity-guided RANSAC framework to sequentially fit each plane instance.
• Score: 37.481945507799594
• License:
• Abstract: This paper presents a generalizable 3D plane detection and reconstruction framework named MonoPlane. Unlike previous robust estimator-based works (which require multiple images or RGB-D input) and learning-based works (which suffer from domain shift), MonoPlane combines the best of two worlds and establishes a plane reconstruction pipeline based on monocular geometric cues, resulting in accurate, robust and scalable 3D plane detection and reconstruction in the wild. Specifically, we first leverage large-scale pre-trained neural networks to obtain the depth and surface normals from a single image. These monocular geometric cues are then incorporated into a proximity-guided RANSAC framework to sequentially fit each plane instance. We exploit effective 3D point proximity and model such proximity via a graph within RANSAC to guide the plane fitting from noisy monocular depths, followed by image-level multi-plane joint optimization to improve the consistency among all plane instances. We further design a simple but effective pipeline to extend this single-view solution to sparse-view 3D plane reconstruction. Extensive experiments on a list of datasets demonstrate our superior zero-shot generalizability over baselines, achieving state-of-the-art plane reconstruction performance in a transferring setting. Our code is available at https://github.com/thuzhaowang/MonoPlane .

Related papers

• UniPlane: Unified Plane Detection and Reconstruction from Posed Monocular Videos [12.328095228008893]
  We present UniPlane, a novel method that unifies plane detection and reconstruction from posed monocular videos. We build a Transformers-based deep neural network that jointly constructs a 3D feature volume for the environment. Experiments on real-world datasets demonstrate that UniPlane outperforms state-of-the-art methods in both plane detection and reconstruction tasks.
  arXiv  Detail & Related papers  (2024-07-04T03:02:27Z)
• GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation [65.33726478659304]
  We introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-06-21T17:49:31Z)
• AirPlanes: Accurate Plane Estimation via 3D-Consistent Embeddings [26.845588648999417]
  We tackle the problem of estimating the planar surfaces in a 3D scene from posed images. We propose a method that predicts multi-view consistent plane embeddings that complement geometry when clustering points into planes. We show through extensive evaluation on the ScanNetV2 dataset that our new method outperforms existing approaches.
  arXiv  Detail & Related papers  (2024-06-13T09:49:31Z)
• Occupancy Planes for Single-view RGB-D Human Reconstruction [120.5818162569105]
  Single-view RGB-D human reconstruction with implicit functions is often formulated as per-point classification. We propose the occupancy planes (OPlanes) representation, which enables to formulate single-view RGB-D human reconstruction as occupancy prediction on planes which slice through the camera's view frustum.
  arXiv  Detail & Related papers  (2022-08-04T17:59:56Z)
• PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos [32.286637700503995]
  PlanarRecon is a framework for globally coherent detection and reconstruction of 3D planes from a posed monocular video. A learning-based tracking and fusion module is designed to merge planes from previous fragments to form a coherent global plane reconstruction. Experiments show that the proposed approach achieves state-of-the-art performances on the ScanNet dataset while being real-time.
  arXiv  Detail & Related papers  (2022-06-15T17:59:16Z)
• PlaneMVS: 3D Plane Reconstruction from Multi-View Stereo [32.81496429134453]
  We present a novel framework named PlaneMVS for 3D plane reconstruction from multiple input views with known camera poses. In contrast, we reconstruct 3D planes with a multi-view-stereo (MVS) pipeline that takes advantage of multi-view geometry. Our method even outperforms a set of SOTA learning-based MVS methods thanks to the learned plane priors.
  arXiv  Detail & Related papers  (2022-03-22T22:35:46Z)
• Monocular Road Planar Parallax Estimation [25.36368935789501]
  Estimating the 3D structure of the drivable surface and surrounding environment is a crucial task for assisted and autonomous driving. We propose Road Planar Parallax Attention Network (RPANet), a new deep neural network for 3D sensing from monocular image sequences. RPANet takes a pair of images aligned by the homography of the road plane as input and outputs a $gamma$ map for 3D reconstruction.
  arXiv  Detail & Related papers  (2021-11-22T10:03:41Z)
• PlaneTR: Structure-Guided Transformers for 3D Plane Recovery [56.23402171871664]
  PlaneTR simultaneously detects and reconstructs planes from a single image. PlaneTR achieves a state-of-the-art performance on the ScanNet and NYUv2 datasets.
  arXiv  Detail & Related papers  (2021-07-27T23:55:40Z)
• Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo [71.59494156155309]
  Existing approaches for multi-view 3D pose estimation explicitly establish cross-view correspondences to group 2D pose detections from multiple camera views. We present our multi-view 3D pose estimation approach based on plane sweep stereo to jointly address the cross-view fusion and 3D pose reconstruction in a single shot.
  arXiv  Detail & Related papers  (2021-04-06T03:49:35Z)
• KAPLAN: A 3D Point Descriptor for Shape Completion [80.15764700137383]
  KAPLAN is a 3D point descriptor that aggregates local shape information via a series of 2D convolutions. In each of those planes, point properties like normals or point-to-plane distances are aggregated into a 2D grid and abstracted into a feature representation with an efficient 2D convolutional encoder. Experiments on public datasets show that KAPLAN achieves state-of-the-art performance for 3D shape completion.
  arXiv  Detail & Related papers  (2020-07-31T21:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.