PlaneFormers: From Sparse View Planes to 3D Reconstruction

• URL: http://arxiv.org/abs/2208.04307v1
• Date: Mon, 8 Aug 2022 17:58:13 GMT
• Title: PlaneFormers: From Sparse View Planes to 3D Reconstruction
• Authors: Samir Agarwala, Linyi Jin, Chris Rockwell, David F. Fouhey
• Abstract summary: We present an approach for the planar surface reconstruction of a scene from images with limited overlap. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning.
• Score: 14.45228936875838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific design decisions are crucial for its success.

Related papers

• Disjoint Pose and Shape for 3D Face Reconstruction [4.096453902709292]
  We propose an end-to-end pipeline that disjointly solves for pose and shape to make the optimization stable and accurate. The proposed method achieves end-to-end topological consistency, enables iterative face pose refinement procedure, and show remarkable improvement on both quantitative and qualitative results.
  arXiv  Detail & Related papers  (2023-08-26T15:18:32Z)
• Neural 3D Scene Reconstruction from Multiple 2D Images without 3D Supervision [41.20504333318276]
  We propose a novel neural reconstruction method that reconstructs scenes using sparse depth under the plane constraints without 3D supervision. We introduce a signed distance function field, a color field, and a probability field to represent a scene. We optimize these fields to reconstruct the scene by using differentiable ray marching with accessible 2D images as supervision.
  arXiv  Detail & Related papers  (2023-06-30T13:30:48Z)
• gSDF: Geometry-Driven Signed Distance Functions for 3D Hand-Object Reconstruction [94.46581592405066]
  We exploit the hand structure and use it as guidance for SDF-based shape reconstruction. We predict kinematic chains of pose transformations and align SDFs with highly-articulated hand poses.
  arXiv  Detail & Related papers  (2023-04-24T10:05:48Z)
• 3D-LatentMapper: View Agnostic Single-View Reconstruction of 3D Shapes [0.0]
  We propose a novel framework that leverages the intermediate latent spaces of Vision Transformer (ViT) and a joint image-text representational model, CLIP, for fast and efficient Single View Reconstruction (SVR) We use the ShapeNetV2 dataset and perform extensive experiments with comparisons to SOTA methods to demonstrate our method's effectiveness.
  arXiv  Detail & Related papers  (2022-12-05T11:45:26Z)
• Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion [54.151979979158085]
  We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available. We leverage an unconditional 3D-aware generator, to which we apply a hybrid inversion scheme where a model produces a first guess of the solution. Our framework can de-render an image in as few as 10 steps, enabling its use in practical scenarios.
  arXiv  Detail & Related papers  (2022-11-21T17:42:42Z)
• Learning Reconstructability for Drone Aerial Path Planning [51.736344549907265]
  We introduce the first learning-based reconstructability predictor to improve view and path planning for large-scale 3D urban scene acquisition using unmanned drones. In contrast to previous approaches, our method learns a model that explicitly predicts how well a 3D urban scene will be reconstructed from a set of viewpoints.
  arXiv  Detail & Related papers  (2022-09-21T08:10:26Z)
• Perspective Reconstruction of Human Faces by Joint Mesh and Landmark Regression [89.8129467907451]
  We propose to simultaneously reconstruct 3D face mesh in the world space and predict 2D face landmarks on the image plane. Based on the predicted 3D and 2D landmarks, the 6DoF (6 Degrees Freedom) face pose can be easily estimated by the solver.
  arXiv  Detail & Related papers  (2022-08-15T12:32:20Z)
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape [77.95154911528365]
  3D Morphable Model (3DMM) fitting has widely benefited face analysis due to its strong 3D priori. Previous reconstructed 3D faces suffer from degraded visual verisimilitude due to the loss of fine-grained geometry. This paper proposes a complete solution to capture the personalized shape so that the reconstructed shape looks identical to the corresponding person.
  arXiv  Detail & Related papers  (2022-04-09T03:46:18Z)
• Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image [32.5277483805739]
  Single-image room layout reconstruction aims to reconstruct the enclosed 3D structure of a room from a single image. This paper considers a more general indoor assumption, i.e., the room layout consists of a single ceiling, a single floor, and several vertical walls.
  arXiv  Detail & Related papers  (2021-04-16T09:24:08Z)
• Adaptive 3D Face Reconstruction from a Single Image [45.736818498242016]
  We propose a novel joint 2D and 3D optimization method to adaptively reconstruct 3D face shapes from a single image. Experimental results on multiple datasets demonstrate that our method can generate high-quality reconstruction from a single color image.
  arXiv  Detail & Related papers  (2020-07-08T09:35:26Z)
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images [61.98279201609436]
  In this paper, we explore a more realistic setup of learning 3D shape from only single-view images. The major difficulty lies in insufficient constraints that can be provided by single view images. We propose an effective adversarial domain confusion method to learn pose-disentangled compact shape space.
  arXiv  Detail & Related papers  (2020-04-03T02:47:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.