Interaction-Driven Active 3D Reconstruction with Object Interiors

• URL: http://arxiv.org/abs/2310.14700v1
• Date: Mon, 23 Oct 2023 08:44:38 GMT
• Title: Interaction-Driven Active 3D Reconstruction with Object Interiors
• Authors: Zihao Yan, Fubao Su, Mingyang Wang, Ruizhen Hu, Hao Zhang, Hui Huang
• Abstract summary: We introduce an active 3D reconstruction method which integrates visual perception, robot-object interaction, and 3D scanning. Our method operates fully automatically by a Fetch robot with built-in RGBD sensors.
• Score: 17.48872400701787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce an active 3D reconstruction method which integrates visual perception, robot-object interaction, and 3D scanning to recover both the exterior and interior, i.e., unexposed, geometries of a target 3D object. Unlike other works in active vision which focus on optimizing camera viewpoints to better investigate the environment, the primary feature of our reconstruction is an analysis of the interactability of various parts of the target object and the ensuing part manipulation by a robot to enable scanning of occluded regions. As a result, an understanding of part articulations of the target object is obtained on top of complete geometry acquisition. Our method operates fully automatically by a Fetch robot with built-in RGBD sensors. It iterates between interaction analysis and interaction-driven reconstruction, scanning and reconstructing detected moveable parts one at a time, where both the articulated part detection and mesh reconstruction are carried out by neural networks. In the final step, all the remaining, non-articulated parts, including all the interior structures that had been exposed by prior part manipulations and subsequently scanned, are reconstructed to complete the acquisition. We demonstrate the performance of our method via qualitative and quantitative evaluation, ablation studies, comparisons to alternatives, as well as experiments in a real environment.

Related papers

• Kinematics-based 3D Human-Object Interaction Reconstruction from Single View [10.684643503514849]
  Existing methods simply predict the body poses merely rely on network training on some indoor datasets. We propose a kinematics-based method that can drive the joints of human body to the human-object contact regions accurately.
  arXiv  Detail & Related papers  (2024-07-19T05:44:35Z)
• Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions [8.059133373836913]
  This paper presents an approach that enables a robot to rapidly learn the complete 3D model of a given object for manipulation in unfamiliar orientations. We use an ensemble of partially constructed NeRF models to quantify model uncertainty to determine the next action. Our approach determines when and how to grasp and re-orient an object given its partial NeRF model and re-estimates the object pose to rectify misalignments introduced during the interaction.
  arXiv  Detail & Related papers  (2024-04-02T10:15:06Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• RoSI: Recovering 3D Shape Interiors from Few Articulation Images [20.430308190444737]
  We present a learning framework to recover the shape interiors of existing 3D models with only their exteriors from multi-view and multi-articulation images. Our neural architecture is trained in a category-agnostic manner and it consists of a motion-aware multi-view analysis phase. In addition, our method also predicts part articulations and is able to realize and even extrapolate the captured motions on the target 3D object.
  arXiv  Detail & Related papers  (2023-04-13T08:45:26Z)
• Neural Part Priors: Learning to Optimize Part-Based Object Completion in RGB-D Scans [27.377128012679076]
  We propose to leverage large-scale synthetic datasets of 3D shapes annotated with part information to learn Neural Part Priors. We can optimize over the learned part priors in order to fit to real-world scanned 3D scenes at test time. Experiments on the ScanNet dataset demonstrate that NPPs significantly outperforms state of the art in part decomposition and object completion.
  arXiv  Detail & Related papers  (2022-03-17T15:05:44Z)
• Towards unconstrained joint hand-object reconstruction from RGB videos [81.97694449736414]
  Reconstructing hand-object manipulations holds a great potential for robotics and learning from human demonstrations. We first propose a learning-free fitting approach for hand-object reconstruction which can seamlessly handle two-hand object interactions.
  arXiv  Detail & Related papers  (2021-08-16T12:26:34Z)
• Object Wake-up: 3-D Object Reconstruction, Animation, and in-situ Rendering from a Single Image [58.69732754597448]
  Given a picture of a chair, could we extract the 3-D shape of the chair, animate its plausible articulations and motions, and render in-situ in its original image space? We devise an automated approach to extract and manipulate articulated objects in single images.
  arXiv  Detail & Related papers  (2021-08-05T16:20:12Z)
• Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments [81.38641691636847]
  We rethink the problem of scene reconstruction from an embodied agent's perspective. We reconstruct an interactive scene using RGB-D data stream. This reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models.
  arXiv  Detail & Related papers  (2021-03-30T05:56:58Z)
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion [78.98074380040838]
  We propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map. Our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.
  arXiv  Detail & Related papers  (2020-06-28T09:50:25Z)
• Learning Unsupervised Hierarchical Part Decomposition of 3D Objects from a Single RGB Image [102.44347847154867]
  We propose a novel formulation that allows to jointly recover the geometry of a 3D object as a set of primitives. Our model recovers the higher level structural decomposition of various objects in the form of a binary tree of primitives. Our experiments on the ShapeNet and D-FAUST datasets demonstrate that considering the organization of parts indeed facilitates reasoning about 3D geometry.
  arXiv  Detail & Related papers  (2020-04-02T17:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.