3D Instance Segmentation Using Deep Learning on RGB-D Indoor Data

• URL: http://arxiv.org/abs/2406.14581v1
• Date: Wed, 19 Jun 2024 08:00:35 GMT
• Title: 3D Instance Segmentation Using Deep Learning on RGB-D Indoor Data
• Authors: Siddiqui Muhammad Yasir, Amin Muhammad Sadiq, Hyunsik Ahn,
• Abstract summary: 2D region based convolutional neural networks (Mask R-CNN) deep learning model with point based rending module is adapted to integrate with depth information to recognize and segment 3D instances of objects. In order to generate 3D point cloud coordinates, segmented 2D pixels of recognized object regions in the RGB image are merged into (u, v) points of the depth image.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D object recognition is a challenging task for intelligent and robot systems in industrial and home indoor environments. It is critical for such systems to recognize and segment the 3D object instances that they encounter on a frequent basis. The computer vision, graphics, and machine learning fields have all given it a lot of attention. Traditionally, 3D segmentation was done with hand-crafted features and designed approaches that did not achieve acceptable performance and could not be generalized to large-scale data. Deep learning approaches have lately become the preferred method for 3D segmentation challenges by their great success in 2D computer vision. However, the task of instance segmentation is currently less explored. In this paper, we propose a novel approach for efficient 3D instance segmentation using red green blue and depth (RGB-D) data based on deep learning. The 2D region based convolutional neural networks (Mask R-CNN) deep learning model with point based rending module is adapted to integrate with depth information to recognize and segment 3D instances of objects. In order to generate 3D point cloud coordinates (x, y, z), segmented 2D pixels (u, v) of recognized object regions in the RGB image are merged into (u, v) points of the depth image. Moreover, we conducted an experiment and analysis to compare our proposed method from various points of view and distances. The experimentation shows the proposed 3D object recognition and instance segmentation are sufficiently beneficial to support object handling in robotic and intelligent systems.

Related papers

• Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models [20.277479473218513]
  We introduce a new task: Zero-Shot 3D Reasoning for parts searching and localization for objects. We design a simple baseline method, Reasoning3D, with the capability to understand and execute complex commands. We show that Reasoning3D can effectively localize and highlight parts of 3D objects based on implicit textual queries.
  arXiv  Detail & Related papers  (2024-05-29T17:56:07Z)
• Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model [108.35777542298224]
  This paper introduces Reason3D, a novel large language model for comprehensive 3D understanding. We propose a hierarchical mask decoder to locate small objects within expansive scenes. Experiments validate that Reason3D achieves remarkable results on large-scale ScanNet and Matterport3D datasets.
  arXiv  Detail & Related papers  (2024-05-27T17:59:41Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation [91.6658845016214]
  We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. We render a 3D shape from multiple views, and set up a dense correspondence learning task within the contrastive learning framework. As a result, the learned 2D representations are view-invariant and geometrically consistent.
  arXiv  Detail & Related papers  (2022-08-18T00:48:15Z)
• PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition [55.38462937452363]
  We propose a unified multi-view cross-modal distillation architecture, including a pretrained deep image encoder as the teacher and a deep point encoder as the student. By pair-wise aligning multi-view visual and geometric descriptors, we can obtain more powerful deep point encoders without exhausting and complicated network modification.
  arXiv  Detail & Related papers  (2022-07-07T07:23:20Z)
• Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection [70.71934539556916]
  We learn geometry-guided depth estimation with projective modeling to advance monocular 3D object detection. Specifically, a principled geometry formula with projective modeling of 2D and 3D depth predictions in the monocular 3D object detection network is devised. Our method remarkably improves the detection performance of the state-of-the-art monocular-based method without extra data by 2.80% on the moderate test setting.
  arXiv  Detail & Related papers  (2021-07-29T12:30:39Z)
• Deep Learning Based 3D Segmentation: A Survey [42.44509605101214]
  3D segmentation is a fundamental and challenging problem in computer vision with applications in autonomous driving and robotics. Deep learning techniques have recently become the tool of choice for 3D segmentation tasks. This paper comprehensively surveys the recent progress in deep learning-based 3D segmentation techniques.
  arXiv  Detail & Related papers  (2021-03-09T13:58:35Z)
• Ground-aware Monocular 3D Object Detection for Autonomous Driving [6.5702792909006735]
  Estimating the 3D position and orientation of objects in the environment with a single RGB camera is a challenging task for low-cost urban autonomous driving and mobile robots. Most of the existing algorithms are based on the geometric constraints in 2D-3D correspondence, which stems from generic 6D object pose estimation. We introduce a novel neural network module to fully utilize such application-specific priors in the framework of deep learning.
  arXiv  Detail & Related papers  (2021-02-01T08:18:24Z)
• Learning Joint 2D-3D Representations for Depth Completion [90.62843376586216]
  We design a simple yet effective neural network block that learns to extract joint 2D and 3D features. Specifically, the block consists of two domain-specific sub-networks that apply 2D convolution on image pixels and continuous convolution on 3D points.
  arXiv  Detail & Related papers  (2020-12-22T22:58:29Z)
• Frustum VoxNet for 3D object detection from RGB-D or Depth images [1.14219428942199]
  We describe a new 3D object detection system from an RGB-D or depth-only point cloud. Our system first detects objects in 2D (either RGB or pseudo-RGB constructed from depth) The main novelty of our system has to do with determining which parts (3D proposals) of the frustums to voxelize.
  arXiv  Detail & Related papers  (2019-10-12T04:06:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.