3D Gated Recurrent Fusion for Semantic Scene Completion

• URL: http://arxiv.org/abs/2002.07269v1
• Date: Mon, 17 Feb 2020 21:45:43 GMT
• Title: 3D Gated Recurrent Fusion for Semantic Scene Completion
• Authors: Yu Liu, Jie Li, Qingsen Yan, Xia Yuan, Chunxia Zhao, Ian Reid and Cesar Cadena
• Abstract summary: This paper tackles the problem of data fusion in the semantic scene completion (SSC) task. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from depth and RGB. Experiments on two benchmark datasets demonstrate the superior performance and the effectiveness of the proposed GRFNet for data fusion in SSC.
• Score: 32.86736222106503
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper tackles the problem of data fusion in the semantic scene completion (SSC) task, which can simultaneously deal with semantic labeling and scene completion. RGB images contain texture details of the object(s) which are vital for semantic scene understanding. Meanwhile, depth images capture geometric clues of high relevance for shape completion. Using both RGB and depth images can further boost the accuracy of SSC over employing one modality in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from depth and RGB by making use of the gate and memory modules. Based on the single-stage fusion, we further propose a multi-stage fusion strategy, which could model the correlations among different stages within the network. Extensive experiments on two benchmark datasets demonstrate the superior performance and the effectiveness of the proposed GRFNet for data fusion in SSC. Code will be made available.

Related papers

• MatchU: Matching Unseen Objects for 6D Pose Estimation from RGB-D Images [57.71600854525037]
  We propose a Fuse-Describe-Match strategy for 6D pose estimation from RGB-D images. MatchU is a generic approach that fuses 2D texture and 3D geometric cues for 6D pose prediction of unseen objects.
  arXiv  Detail & Related papers  (2024-03-03T14:01:03Z)
• FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything [1.5728609542259502]
  This paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation.
  arXiv  Detail & Related papers  (2024-02-29T22:59:27Z)
• Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images [11.100398985633754]
  We propose an end-to-end framework for recovering dense meshes for both hands. Our framework employs ResNet50 and PointNet++ to derive features from RGB and point cloud. We also introduce a novel pyramid deep fusion network (PDFNet) to aggregate features at different scales.
  arXiv  Detail & Related papers  (2023-07-12T09:33:21Z)
• Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution [123.04455334124188]
  Guided depth map super-resolution (GDSR) aims to upsample low-resolution (LR) depth maps with additional information involved in high-resolution (HR) RGB images from the same scene. In this paper, we propose the Spherical Space feature Decomposition Network (SSDNet) to solve the above issues. Our method can achieve state-of-the-art results on four test datasets, as well as successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2023-03-15T21:22:21Z)
• Similarity-Aware Fusion Network for 3D Semantic Segmentation [87.51314162700315]
  We propose a similarity-aware fusion network (SAFNet) to adaptively fuse 2D images and 3D point clouds for 3D semantic segmentation. We employ a late fusion strategy where we first learn the geometric and contextual similarities between the input and back-projected (from 2D pixels) point clouds. We show that SAFNet significantly outperforms existing state-of-the-art fusion-based approaches across various data integrity.
  arXiv  Detail & Related papers  (2021-07-04T09:28:18Z)
• Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images [69.5662419067878]
  Grounding referring expressions in RGBD image has been an emerging field. We present a novel task of 3D visual grounding in single-view RGBD image where the referred objects are often only partially scanned due to occlusion. Our approach first fuses the language and the visual features at the bottom level to generate a heatmap that localizes the relevant regions in the RGBD image. Then our approach conducts an adaptive feature learning based on the heatmap and performs the object-level matching with another visio-linguistic fusion to finally ground the referred object.
  arXiv  Detail & Related papers  (2021-03-14T11:18:50Z)
• Self-Supervised Representation Learning for RGB-D Salient Object Detection [93.17479956795862]
  We use Self-Supervised Representation Learning to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and un RGB-D datasets to perform pre-training, which make the network capture rich semantic contexts. For the inherent problem of cross-modal fusion in RGB-D SOD, we propose a multi-path fusion module.
  arXiv  Detail & Related papers  (2021-01-29T09:16:06Z)
• Cross-Modality 3D Object Detection [63.29935886648709]
  We present a novel two-stage multi-modal fusion network for 3D object detection. The whole architecture facilitates two-stage fusion. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.
  arXiv  Detail & Related papers  (2020-08-16T11:01:20Z)
• Deep Convolutional Sparse Coding Networks for Image Fusion [29.405149234582623]
  Deep learning has emerged as an important tool for image fusion. This paper presents three deep convolutional sparse coding (CSC) networks for three kinds of image fusion tasks.
  arXiv  Detail & Related papers  (2020-05-18T04:12:01Z)
• Attention-based Multi-modal Fusion Network for Semantic Scene Completion [35.93265545962268]
  This paper presents an end-to-end 3D convolutional network named attention-based multi-modal fusion network (AMFNet) for the semantic scene completion (SSC) task. Compared with previous methods which use only the semantic features extracted from RGB-D images, the proposed AMFNet learns to perform effective 3D scene completion and semantic segmentation simultaneously. It is achieved by employing a multi-modal fusion architecture boosted from 2D semantic segmentation and a 3D semantic completion network empowered by residual attention blocks.
  arXiv  Detail & Related papers  (2020-03-31T02:00:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.