FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything

• URL: http://arxiv.org/abs/2403.00175v2
• Date: Wed, 1 May 2024 12:34:53 GMT
• Title: FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything
• Authors: Safouane El Ghazouali, Youssef Mhirit, Ali Oukhrid, Umberto Michelucci, Hichem Nouira,
• Abstract summary: This paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation.
• Score: 1.5728609542259502
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In the realm of computer vision, the integration of advanced techniques into the processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth map as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color \textit{RGB} and depth \textit{D} channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation. The code and pre-trained models are publicly available at https://github.com/safouaneelg/FusionVision/.

Related papers

• ViDSOD-100: A New Dataset and a Baseline Model for RGB-D Video Salient Object Detection [51.16181295385818]
  We first collect an annotated RGB-D video SODOD (DSOD-100) dataset, which contains 100 videos within a total of 9,362 frames. All the frames in each video are manually annotated to a high-quality saliency annotation. We propose a new baseline model, named attentive triple-fusion network (ATF-Net) for RGB-D salient object detection.
  arXiv  Detail & Related papers  (2024-06-18T12:09:43Z)
• Enhanced Automotive Object Detection via RGB-D Fusion in a DiffusionDet Framework [0.0]
  Vision-based autonomous driving requires reliable and efficient object detection. This work proposes a DiffusionDet-based framework that exploits data fusion from the monocular camera and depth sensor to provide the RGB and depth (RGB-D) data. By integrating the textural and color features from RGB images with the spatial depth information from the LiDAR sensors, the proposed framework employs a feature fusion that substantially enhances object detection of automotive targets.
  arXiv  Detail & Related papers  (2024-06-05T10:24:00Z)
• EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion [55.367269556557645]
  EvPlug learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model. We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation.
  arXiv  Detail & Related papers  (2023-12-28T10:05:13Z)
• FusionViT: Hierarchical 3D Object Detection via LiDAR-Camera Vision Transformer Fusion [8.168523242105763]
  We will introduce a novel vision transformer-based 3D object detection model, namely FusionViT. Our FusionViT model can achieve state-of-the-art performance and outperforms existing baseline methods.
  arXiv  Detail & Related papers  (2023-11-07T00:12:01Z)
• Anyview: Generalizable Indoor 3D Object Detection with Variable Frames [63.51422844333147]
  We present a novel 3D detection framework named AnyView for our practical applications. Our method achieves both great generalizability and high detection accuracy with a simple and clean architecture.
  arXiv  Detail & Related papers  (2023-10-09T02:15:45Z)
• DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention [50.11672196146829]
  3D object detection with surround-view images is an essential task for autonomous driving. We propose DETR4D, a Transformer-based framework that explores sparse attention and direct feature query for 3D object detection in multi-view images.
  arXiv  Detail & Related papers  (2022-12-15T14:18:47Z)
• CMR3D: Contextualized Multi-Stage Refinement for 3D Object Detection [57.44434974289945]
  We propose Contextualized Multi-Stage Refinement for 3D Object Detection (CMR3D) framework. Our framework takes a 3D scene as input and strives to explicitly integrate useful contextual information of the scene. In addition to 3D object detection, we investigate the effectiveness of our framework for the problem of 3D object counting.
  arXiv  Detail & Related papers  (2022-09-13T05:26:09Z)
• ODAM: Object Detection, Association, and Mapping using Posed RGB Video [36.16010611723447]
  We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN)
  arXiv  Detail & Related papers  (2021-08-23T13:28:10Z)
• Perception-aware Multi-sensor Fusion for 3D LiDAR Semantic Segmentation [59.42262859654698]
  3D semantic segmentation is important in scene understanding for many applications, such as auto-driving and robotics. Existing fusion-based methods may not achieve promising performance due to vast difference between two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to exploit perceptual information from two modalities.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• A Multi-Level Approach to Waste Object Segmentation [10.20384144853726]
  We address the problem of localizing waste objects from a color image and an optional depth image. Our method integrates the intensity and depth information at multiple levels of spatial granularity. We create a new RGBD waste object segmentation, MJU-Waste, that is made public to facilitate future research in this area.
  arXiv  Detail & Related papers  (2020-07-08T16:49:25Z)
• 3D Gated Recurrent Fusion for Semantic Scene Completion [32.86736222106503]
  This paper tackles the problem of data fusion in the semantic scene completion (SSC) task. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from depth and RGB. Experiments on two benchmark datasets demonstrate the superior performance and the effectiveness of the proposed GRFNet for data fusion in SSC.
  arXiv  Detail & Related papers  (2020-02-17T21:45:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.