Explicitly Guided Information Interaction Network for Cross-modal Point Cloud Completion

• URL: http://arxiv.org/abs/2407.02887v3
• Date: Tue, 23 Jul 2024 03:03:54 GMT
• Title: Explicitly Guided Information Interaction Network for Cross-modal Point Cloud Completion
• Authors: Hang Xu, Chen Long, Wenxiao Zhang, Yuan Liu, Zhen Cao, Zhen Dong, Bisheng Yang,
• Abstract summary: We introduce EGIInet (Explicitly Guided Information Interaction Network), a model for View-guided Point cloud Completion task. EGIInet efficiently combines the information from two modalities by leveraging the geometric nature of the completion task. We propose a novel explicitly guided information interaction strategy that could help the network identify critical information within images.
• Score: 34.102157812175854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we explore a novel framework, EGIInet (Explicitly Guided Information Interaction Network), a model for View-guided Point cloud Completion (ViPC) task, which aims to restore a complete point cloud from a partial one with a single view image. In comparison with previous methods that relied on the global semantics of input images, EGIInet efficiently combines the information from two modalities by leveraging the geometric nature of the completion task. Specifically, we propose an explicitly guided information interaction strategy supported by modal alignment for point cloud completion. First, in contrast to previous methods which simply use 2D and 3D backbones to encode features respectively, we unified the encoding process to promote modal alignment. Second, we propose a novel explicitly guided information interaction strategy that could help the network identify critical information within images, thus achieving better guidance for completion. Extensive experiments demonstrate the effectiveness of our framework, and we achieved a new state-of-the-art (+16% CD over XMFnet) in benchmark datasets despite using fewer parameters than the previous methods. The pre-trained model and code and are available at https://github.com/WHU-USI3DV/EGIInet.

Related papers

• Point-In-Context: Understanding Point Cloud via In-Context Learning [67.20277182808992]
  We introduce Point-In-Context (PIC), a novel framework for 3D point cloud understanding via in-context learning. We address the technical challenge of effectively extending masked point modeling to 3D point clouds by introducing a Joint Sampling module. We propose two novel training strategies, In-Context Labeling and In-Context Enhancing, forming an extended version of PIC named Point-In-Context-Segmenter (PIC-S)
  arXiv  Detail & Related papers  (2024-04-18T17:32:32Z)
• Cross-Modal Information-Guided Network using Contrastive Learning for Point Cloud Registration [17.420425069785946]
  We present a novel Cross-Modal Information-Guided Network (CMIGNet) for point cloud registration. We first incorporate the projected images from the point clouds and fuse the cross-modal features using the attention mechanism. We employ two contrastive learning strategies, namely overlapping contrastive learning and cross-modal contrastive learning.
  arXiv  Detail & Related papers  (2023-11-02T12:56:47Z)
• Fine-grained Text and Image Guided Point Cloud Completion with CLIP Model [15.625396852353655]
  We propose a novel multimodal fusion network for point cloud completion. We employ a pre-trained vision-language model that is trained with a large amount of image-text pairs. To further explore the effectiveness of fine-grained text descriptions for point cloud completion, we also build a text corpus with fine-grained descriptions.
  arXiv  Detail & Related papers  (2023-08-17T03:05:18Z)
• Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysis [43.13887916301742]
  This paper introduces a simple but effective point cloud cross-modality training (PointCMT) strategy to boost point cloud analysis. To effectively acquire auxiliary knowledge from view images, we develop a teacher-student framework and formulate the cross modal learning as a knowledge distillation problem. We verify significant gains on various datasets using appealing backbones, i.e., equipped with PointCMT, PointNet++ and PointMLP.
  arXiv  Detail & Related papers  (2022-10-09T09:35:22Z)
• Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning [59.64695628433855]
  We propose a novel cross-modality weakly supervised method for 3D segmentation, incorporating complementary information from unlabeled images. Basically, we design a dual-branch network equipped with an active labeling strategy, to maximize the power of tiny parts of labels. Our method even outperforms the state-of-the-art fully supervised competitors with less than 1% actively selected annotations.
  arXiv  Detail & Related papers  (2022-09-16T07:59:04Z)
• GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation [91.15865862160088]
  We introduce a geometric flow network (GFNet) to explore the geometric correspondence between different views in an align-before-fuse manner. Specifically, we devise a novel geometric flow module (GFM) to bidirectionally align and propagate the complementary information across different views.
  arXiv  Detail & Related papers  (2022-07-06T11:48:08Z)
• Semantic keypoint-based pose estimation from single RGB frames [64.80395521735463]
  We present an approach to estimating the continuous 6-DoF pose of an object from a single RGB image. The approach combines semantic keypoints predicted by a convolutional network (convnet) with a deformable shape model. We show that our approach can accurately recover the 6-DoF object pose for both instance- and class-based scenarios.
  arXiv  Detail & Related papers  (2022-04-12T15:03:51Z)
• View-Guided Point Cloud Completion [43.139758470826806]
  ViPC (view-guided point cloud completion) takes the missing crucial global structure information from an extra single-view image. Our method achieves significantly superior results over typical existing solutions on a new large-scale dataset.
  arXiv  Detail & Related papers  (2021-04-12T17:35:45Z)
• PC-RGNN: Point Cloud Completion and Graph Neural Network for 3D Object Detection [57.49788100647103]
  LiDAR-based 3D object detection is an important task for autonomous driving. Current approaches suffer from sparse and partial point clouds of distant and occluded objects. In this paper, we propose a novel two-stage approach, namely PC-RGNN, dealing with such challenges by two specific solutions.
  arXiv  Detail & Related papers  (2020-12-18T18:06:43Z)
• Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
  We propose to adopt the graph propagation to capture the observed spatial contexts. We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
  arXiv  Detail & Related papers  (2020-08-25T06:00:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.