Robust Shape Fitting for 3D Scene Abstraction

• URL: http://arxiv.org/abs/2403.10452v1
• Date: Fri, 15 Mar 2024 16:37:43 GMT
• Title: Robust Shape Fitting for 3D Scene Abstraction
• Authors: Florian Kluger, Eric Brachmann, Michael Ying Yang, Bodo Rosenhahn,
• Abstract summary: In particular, we can describe man-made environments using volumetric primitives such as cuboids or cylinders. We propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.
• Score: 33.84212609361491
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans perceive and construct the world as an arrangement of simple parametric models. In particular, we can often describe man-made environments using volumetric primitives such as cuboids or cylinders. Inferring these primitives is important for attaining high-level, abstract scene descriptions. Previous approaches for primitive-based abstraction estimate shape parameters directly and are only able to reproduce simple objects. In contrast, we propose a robust estimator for primitive fitting, which meaningfully abstracts complex real-world environments using cuboids. A RANSAC estimator guided by a neural network fits these primitives to a depth map. We condition the network on previously detected parts of the scene, parsing it one-by-one. To obtain cuboids from single RGB images, we additionally optimise a depth estimation CNN end-to-end. Naively minimising point-to-primitive distances leads to large or spurious cuboids occluding parts of the scene. We thus propose an improved occlusion-aware distance metric correctly handling opaque scenes. Furthermore, we present a neural network based cuboid solver which provides more parsimonious scene abstractions while also reducing inference time. The proposed algorithm does not require labour-intensive labels, such as cuboid annotations, for training. Results on the NYU Depth v2 dataset demonstrate that the proposed algorithm successfully abstracts cluttered real-world 3D scene layouts.

Related papers

• N-BVH: Neural ray queries with bounding volume hierarchies [51.430495562430565]
  In 3D computer graphics, the bulk of a scene's memory usage is due to polygons and textures. We devise N-BVH, a neural compression architecture designed to answer arbitrary ray queries in 3D. Our method provides faithful approximations of visibility, depth, and appearance attributes.
  arXiv  Detail & Related papers  (2024-05-25T13:54:34Z)
• ALSO: Automotive Lidar Self-supervision by Occupancy estimation [70.70557577874155]
  We propose a new self-supervised method for pre-training the backbone of deep perception models operating on point clouds. The core idea is to train the model on a pretext task which is the reconstruction of the surface on which the 3D points are sampled. The intuition is that if the network is able to reconstruct the scene surface, given only sparse input points, then it probably also captures some fragments of semantic information.
  arXiv  Detail & Related papers  (2022-12-12T13:10:19Z)
• ShAPO: Implicit Representations for Multi-Object Shape, Appearance, and Pose Optimization [40.36229450208817]
  We present ShAPO, a method for joint multi-object detection, 3D textured reconstruction, 6D object pose and size estimation. Key to ShAPO is a single-shot pipeline to regress shape, appearance and pose latent codes along with the masks of each object instance. Our method significantly out-performs all baselines on the NOCS dataset with an 8% absolute improvement in mAP for 6D pose estimation.
  arXiv  Detail & Related papers  (2022-07-27T17:59:31Z)
• Neural 3D Scene Reconstruction with the Manhattan-world Assumption [58.90559966227361]
  This paper addresses the challenge of reconstructing 3D indoor scenes from multi-view images. Planar constraints can be conveniently integrated into the recent implicit neural representation-based reconstruction methods. The proposed method outperforms previous methods by a large margin on 3D reconstruction quality.
  arXiv  Detail & Related papers  (2022-05-05T17:59:55Z)
• Cuboids Revisited: Learning Robust 3D Shape Fitting to Single RGB Images [44.223070672713455]
  In particular, man-made environments commonly consist of volumetric primitives such as cuboids or cylinders. Previous approaches directly estimate shape parameters from a 2D or 3D input, and are only able to reproduce simple objects. We propose a robust estimator for primitive fitting, which can meaningfully abstract real-world environments using cuboids.
  arXiv  Detail & Related papers  (2021-05-05T13:36:00Z)
• Neural Parts: Learning Expressive 3D Shape Abstractions with Invertible Neural Networks [118.20778308823779]
  We present a novel 3D primitive representation that defines primitives using an Invertible Neural Network (INN) Our model learns to parse 3D objects into semantically consistent part arrangements without any part-level supervision.
  arXiv  Detail & Related papers  (2021-03-18T17:59:31Z)
• Rapid Pose Label Generation through Sparse Representation of Unknown Objects [7.32172860877574]
  This work presents an approach for rapidly generating real-world, pose-annotated RGB-D data for unknown objects. We first source minimalistic labelings of an ordered set of arbitrarily chosen keypoints over a set of RGB-D videos. By solving an optimization problem, we combine these labels under a world frame to recover a sparse, keypoint-based representation of the object.
  arXiv  Detail & Related papers  (2020-11-07T15:14:03Z)
• Convolutional Occupancy Networks [88.48287716452002]
  We propose Convolutional Occupancy Networks, a more flexible implicit representation for detailed reconstruction of objects and 3D scenes. By combining convolutional encoders with implicit occupancy decoders, our model incorporates inductive biases, enabling structured reasoning in 3D space. We empirically find that our method enables the fine-grained implicit 3D reconstruction of single objects, scales to large indoor scenes, and generalizes well from synthetic to real data.
  arXiv  Detail & Related papers  (2020-03-10T10:17:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.