Robust Bayesian Scene Reconstruction by Leveraging Retrieval-Augmented Priors

• URL: http://arxiv.org/abs/2411.19461v2
• Date: Sun, 08 Dec 2024 01:04:18 GMT
• Title: Robust Bayesian Scene Reconstruction by Leveraging Retrieval-Augmented Priors
• Authors: Herbert Wright, Weiming Zhi, Matthew Johnson-Roberson, Tucker Hermans,
• Abstract summary: Building 3D representations of object geometry is critical for many downstream robotics tasks.<n>In this work, we focus on the problem of reconstructing a multi-object scene from a single RGBD image.<n>We propose BRRP, a reconstruction method that leverages preexisting mesh datasets to build an informative prior.
• Score: 17.05305897044699
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Constructing 3D representations of object geometry is critical for many downstream robotics tasks, particularly tabletop manipulation problems. These representations must be built from potentially noisy partial observations. In this work, we focus on the problem of reconstructing a multi-object scene from a single RGBD image, generally from a fixed camera in the scene. Traditional scene representation methods generally cannot infer the geometry of unobserved regions of the objects from the image. Attempts have been made to leverage deep learning to train on a dataset of observed objects and representations, and then generalize to new observations. However, this can be brittle to noisy real-world observations and objects not contained in the dataset, and cannot reason about their confidence. We propose BRRP, a reconstruction method that leverages preexisting mesh datasets to build an informative prior during robust probabilistic reconstruction. In order to make our method more efficient, we introduce the concept of retrieval-augmented prior, where we retrieve relevant components of our prior distribution during inference. The prior is used to estimate the geometry of occluded portions of the in-scene objects. Our method produces a distribution over object shape that can be used for reconstruction or measuring uncertainty. We evaluate our method in both simulated scenes and in the real world. We demonstrate the robustness of our method against deep learning-only approaches while being more accurate than a method without an informative prior.

Related papers

• ObjectCarver: Semi-automatic segmentation, reconstruction and separation of 3D objects [44.38881095466177]
  Implicit neural fields have made remarkable progress in reconstructing 3D surfaces from multiple images. Previous work has attempted to tackle this problem by introducing a framework to train separate signed distance fields. We introduce our method, ObjectCarver, to tackle the problem of object separation from just click input in a single view.
  arXiv  Detail & Related papers  (2024-07-26T22:13:20Z)
• DVMNet++: Rethinking Relative Pose Estimation for Unseen Objects [59.51874686414509]
  Existing approaches typically predict 3D translation utilizing the ground-truth object bounding box and approximate 3D rotation with a large number of discrete hypotheses. We present a Deep Voxel Matching Network (DVMNet++) that computes the relative object pose in a single pass. Our approach delivers more accurate relative pose estimates for novel objects at a lower computational cost compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-20T15:41:32Z)
• ICGNet: A Unified Approach for Instance-Centric Grasping [42.92991092305974]
  We introduce an end-to-end architecture for object-centric grasping. We show the effectiveness of the proposed method by extensively evaluating it against state-of-the-art methods on synthetic datasets.
  arXiv  Detail & Related papers  (2024-01-18T12:41:41Z)
• What You See Is What You Detect: Towards better Object Densification in 3D detection [2.3436632098950456]
  The widely-used full-shape completion approach actually leads to a higher error-upper bound especially for far away objects and small objects like pedestrians. We introduce a visible part completion method that requires only 11.3% of the prediction points that previous methods generate. To recover the dense representation, we propose a mesh-deformation-based method to augment the point set associated with visible foreground objects.
  arXiv  Detail & Related papers  (2023-10-27T01:46:37Z)
• A Fusion of Variational Distribution Priors and Saliency Map Replay for Continual 3D Reconstruction [1.2289361708127877]
  Single-image 3D reconstruction is a research challenge focused on predicting 3D object shapes from single-view images. This task requires significant data acquisition to predict both visible and occluded portions of the shape. We propose a continual learning-based 3D reconstruction method where our goal is to design a model using Variational Priors that can still reconstruct the previously seen classes reasonably even after training on new classes.
  arXiv  Detail & Related papers  (2023-08-17T06:48:55Z)
• Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior [9.716201630968433]
  3D object reconstruction is important for semantic scene understanding. It is challenging to reconstruct detailed 3D shapes from monocular images directly due to a lack of depth information, occlusion and noise. We tackle this problem by leveraging a neural object representation which learns an object shape distribution from large dataset of 3d object models and maps it into a latent space. We propose a method to model uncertainty as part of the representation and define an uncertainty-aware encoder which generates latent codes with uncertainty directly from individual input images.
  arXiv  Detail & Related papers  (2023-06-17T03:25:13Z)
• Neural Kernel Surface Reconstruction [80.51581494300423]
  We present a novel method for reconstructing a 3D implicit surface from a large-scale, sparse, and noisy point cloud. Our approach builds upon the recently introduced Neural Kernel Fields representation.
  arXiv  Detail & Related papers  (2023-05-31T06:25:18Z)
• BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects [89.2314092102403]
  We present a near real-time method for 6-DoF tracking of an unknown object from a monocular RGBD video sequence. Our method works for arbitrary rigid objects, even when visual texture is largely absent.
  arXiv  Detail & Related papers  (2023-03-24T17:13:49Z)
• Reconstructing Hand-Held Objects from Monocular Video [95.06750686508315]
  This paper presents an approach that reconstructs a hand-held object from a monocular video. In contrast to many recent methods that directly predict object geometry by a trained network, the proposed approach does not require any learned prior to the object.
  arXiv  Detail & Related papers  (2022-11-30T09:14:58Z)
• LocPoseNet: Robust Location Prior for Unseen Object Pose Estimation [69.70498875887611]
  LocPoseNet is able to robustly learn location prior for unseen objects. Our method outperforms existing works by a large margin on LINEMOD and GenMOP.
  arXiv  Detail & Related papers  (2022-11-29T15:21:34Z)
• Semantic keypoint-based pose estimation from single RGB frames [64.80395521735463]
  We present an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. We show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios.
  arXiv  Detail & Related papers  (2022-04-12T15:03:51Z)
• IFOR: Iterative Flow Minimization for Robotic Object Rearrangement [92.97142696891727]
  IFOR, Iterative Flow Minimization for Robotic Object Rearrangement, is an end-to-end method for the problem of object rearrangement for unknown objects. We show that our method applies to cluttered scenes, and in the real world, while training only on synthetic data.
  arXiv  Detail & Related papers  (2022-02-01T20:03:56Z)
• Holistic 3D Scene Understanding from a Single Image with Implicit Representation [112.40630836979273]
  We present a new pipeline for holistic 3D scene understanding from a single image. We propose an image-based local structured implicit network to improve the object shape estimation. We also refine 3D object pose and scene layout via a novel implicit scene graph neural network.
  arXiv  Detail & Related papers  (2021-03-11T02:52:46Z)
• Next-best-view Regression using a 3D Convolutional Neural Network [0.9449650062296823]
  We propose a data-driven approach to address the next-best-view problem. The proposed approach trains a 3D convolutional neural network with previous reconstructions in order to regress the btxtposition of the next-best-view. We have validated the proposed approach making use of two groups of experiments.
  arXiv  Detail & Related papers  (2021-01-23T01:50:26Z)
• Single View Metrology in the Wild [94.7005246862618]
  We present a novel approach to single view metrology that can recover the absolute scale of a scene represented by 3D heights of objects or camera height above the ground. Our method relies on data-driven priors learned by a deep network specifically designed to imbibe weakly supervised constraints from the interplay of the unknown camera with 3D entities such as object heights. We demonstrate state-of-the-art qualitative and quantitative results on several datasets as well as applications including virtual object insertion.
  arXiv  Detail & Related papers  (2020-07-18T22:31:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.