Weakly-Supervised End-to-End CAD Retrieval to Scan Objects

• URL: http://arxiv.org/abs/2203.12873v1
• Date: Thu, 24 Mar 2022 06:30:47 GMT
• Title: Weakly-Supervised End-to-End CAD Retrieval to Scan Objects
• Authors: Tim Beyer, Angela Dai
• Abstract summary: We propose a new weakly-supervised approach to retrieve semantically and structurally similar CAD models to a query 3D scanned scene. Our approach leverages a fully-differentiable top-$k$ retrieval layer, enabling end-to-end training guided by geometric and perceptual similarity of the top retrieved CAD models to the scan queries.
• Score: 25.41908065938424
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: CAD model retrieval to real-world scene observations has shown strong promise as a basis for 3D perception of objects and a clean, lightweight mesh-based scene representation; however, current approaches to retrieve CAD models to a query scan rely on expensive manual annotations of 1:1 associations of CAD-scan objects, which typically contain strong lower-level geometric differences. We thus propose a new weakly-supervised approach to retrieve semantically and structurally similar CAD models to a query 3D scanned scene without requiring any CAD-scan associations, and only object detection information as oriented bounding boxes. Our approach leverages a fully-differentiable top-$k$ retrieval layer, enabling end-to-end training guided by geometric and perceptual similarity of the top retrieved CAD models to the scan queries. We demonstrate that our weakly-supervised approach can outperform fully-supervised retrieval methods on challenging real-world ScanNet scans, and maintain robustness for unseen class categories, achieving significantly improved performance over fully-supervised state of the art in zero-shot CAD retrieval.

Related papers

• LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset [17.530432165466507]
  We present a novel Cross-Modal Shape Reconstruction (DisCo) method and an Occupancy-Guided 3D Object Detection (OccGOD) method. Our methods achieve state-of-the-art performance in both instance-level scene reconstruction and 3D object detection tasks.
  arXiv  Detail & Related papers  (2023-12-19T18:50:10Z)
• DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image [34.47379913018661]
  We propose DiffCAD, the first weakly-supervised probabilistic approach to CAD retrieval and alignment from an RGB image. We formulate this as a conditional generative task, leveraging diffusion to learn implicit probabilistic models capturing the shape, pose, and scale of CAD objects in an image. Our approach is trained only on synthetic data, leveraging monocular depth and mask estimates to enable robust zero-shot adaptation to various real target domains.
  arXiv  Detail & Related papers  (2023-11-30T15:10:21Z)
• Sparse Multi-Object Render-and-Compare [33.97243145891282]
  Reconstructing 3D shape and pose of static objects from a single image is an essential task for various industries. Directly predicting 3D shapes produces unrealistic, overly smoothed or tessellated shapes. Retrieving CAD models ensures realistic shapes but requires robust and accurate alignment.
  arXiv  Detail & Related papers  (2023-10-17T12:01:32Z)
• NeurOCS: Neural NOCS Supervision for Monocular 3D Object Localization [80.3424839706698]
  We present NeurOCS, a framework that uses instance masks 3D boxes as input to learn 3D object shapes by means of differentiable rendering. Our approach rests on insights in learning a category-level shape prior directly from real driving scenes. We make critical design choices to learn object coordinates more effectively from an object-centric view.
  arXiv  Detail & Related papers  (2023-05-28T16:18:41Z)
• Multiview Compressive Coding for 3D Reconstruction [77.95706553743626]
  We introduce a simple framework that operates on 3D points of single objects or whole scenes. Our model, Multiview Compressive Coding, learns to compress the input appearance and geometry to predict the 3D structure.
  arXiv  Detail & Related papers  (2023-01-19T18:59:52Z)
• Stereo Neural Vernier Caliper [57.187088191829886]
  We propose a new object-centric framework for learning-based stereo 3D object detection. We tackle a problem of how to predict a refined update given an initial 3D cuboid guess. Our approach achieves state-of-the-art performance on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-03-21T14:36:07Z)
• Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image [58.953160501596805]
  We propose a novel approach towards constructing a joint embedding space between 2D images and 3D CAD models in a patch-wise fashion. Our approach is more robust than state of the art in real-world scenarios without any exact CAD matches.
  arXiv  Detail & Related papers  (2021-08-20T20:58:52Z)
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection [138.2892824662943]
  A promising solution is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. Recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective.
  arXiv  Detail & Related papers  (2021-08-17T17:56:12Z)
• Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve [54.054575408582565]
  We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image. We present Mask2CAD, which jointly detects objects in real-world images and for each detected object, optimize for the most similar CAD model and its pose. This produces a clean, lightweight representation of the objects in an image.
  arXiv  Detail & Related papers  (2020-07-26T00:08:37Z)
• CAD-Deform: Deformable Fitting of CAD Models to 3D Scans [30.451330075135076]
  We introduce CAD-Deform, a method which obtains more accurate CAD-to-scan fits by non-rigidly deforming retrieved CAD models. A series of experiments demonstrate that our method achieves significantly tighter scan-to-CAD fits, allowing a more accurate digital replica of the scanned real-world environment.
  arXiv  Detail & Related papers  (2020-07-23T12:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.