Camera-LiDAR Cross-modality Gait Recognition

• URL: http://arxiv.org/abs/2407.02038v3
• Date: Thu, 4 Jul 2024 05:02:06 GMT
• Title: Camera-LiDAR Cross-modality Gait Recognition
• Authors: Wenxuan Guo, Yingping Liang, Zhiyu Pan, Ziheng Xi, Jianjiang Feng, Jie Zhou,
• Abstract summary: We propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. To the best of our knowledge, this is the first work to address cross-modality gait recognition.
• Score: 29.694346498355443
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Gait recognition is a crucial biometric identification technique. Camera-based gait recognition has been widely applied in both research and industrial fields. LiDAR-based gait recognition has also begun to evolve most recently, due to the provision of 3D structural information. However, in certain applications, cameras fail to recognize persons, such as in low-light environments and long-distance recognition scenarios, where LiDARs work well. On the other hand, the deployment cost and complexity of LiDAR systems limit its wider application. Therefore, it is essential to consider cross-modality gait recognition between cameras and LiDARs for a broader range of applications. In this work, we propose the first cross-modality gait recognition framework between Camera and LiDAR, namely CL-Gait. It employs a two-stream network for feature embedding of both modalities. This poses a challenging recognition task due to the inherent matching between 3D and 2D data, exhibiting significant modality discrepancy. To align the feature spaces of the two modalities, i.e., camera silhouettes and LiDAR points, we propose a contrastive pre-training strategy to mitigate modality discrepancy. To make up for the absence of paired camera-LiDAR data for pre-training, we also introduce a strategy for generating data on a large scale. This strategy utilizes monocular depth estimated from single RGB images and virtual cameras to generate pseudo point clouds for contrastive pre-training. Extensive experiments show that the cross-modality gait recognition is very challenging but still contains potential and feasibility with our proposed model and pre-training strategy. To the best of our knowledge, this is the first work to address cross-modality gait recognition.

Related papers

• What Really Matters for Learning-based LiDAR-Camera Calibration [50.2608502974106]
  This paper revisits the development of learning-based LiDAR-Camera calibration. We identify the critical limitations of regression-based methods with the widely used data generation pipeline. We also investigate how the input data format and preprocessing operations impact network performance.
  arXiv  Detail & Related papers  (2025-01-28T14:12:32Z)
• GSPR: Multimodal Place Recognition Using 3D Gaussian Splatting for Autonomous Driving [9.023864430027333]
  multimodal place recognition has gained increasing attention due to their ability to overcome weaknesses of uni sensor systems. We propose a 3D Gaussian-based multimodal place recognition neural network dubbed GSPR.
  arXiv  Detail & Related papers  (2024-10-01T00:43:45Z)
• Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving [58.16024314532443]
  We introduce LaserMix++, a framework that integrates laser beam manipulations from disparate LiDAR scans and incorporates LiDAR-camera correspondences to assist data-efficient learning. Results demonstrate that LaserMix++ outperforms fully supervised alternatives, achieving comparable accuracy with five times fewer annotations. This substantial advancement underscores the potential of semi-supervised approaches in reducing the reliance on extensive labeled data in LiDAR-based 3D scene understanding systems.
  arXiv  Detail & Related papers  (2024-05-08T17:59:53Z)
• ODM3D: Alleviating Foreground Sparsity for Semi-Supervised Monocular 3D Object Detection [15.204935788297226]
  ODM3D framework entails cross-modal knowledge distillation at various levels to inject LiDAR-domain knowledge into a monocular detector during training. By identifying foreground sparsity as the main culprit behind existing methods' suboptimal training, we exploit the precise localisation information embedded in LiDAR points. Our method ranks 1st in both KITTI validation and test benchmarks, significantly surpassing all existing monocular methods, supervised or semi-supervised.
  arXiv  Detail & Related papers  (2023-10-28T07:12:09Z)
• Egocentric RGB+Depth Action Recognition in Industry-Like Settings [50.38638300332429]
  Our work focuses on recognizing actions from egocentric RGB and Depth modalities in an industry-like environment. Our framework is based on the 3D Video SWIN Transformer to encode both RGB and Depth modalities effectively. Our method also secured first place at the multimodal action recognition challenge at ICIAP 2023.
  arXiv  Detail & Related papers  (2023-09-25T08:56:22Z)
• Geometric-aware Pretraining for Vision-centric 3D Object Detection [77.7979088689944]
  We propose a novel geometric-aware pretraining framework called GAPretrain. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively.
  arXiv  Detail & Related papers  (2023-04-06T14:33:05Z)
• Gait Recognition in Large-scale Free Environment via Single LiDAR [35.684257181154905]
  LiDAR's ability to capture depth makes it pivotal for robotic perception and holds promise for real-world gait recognition. We present the Hierarchical Multi-representation Feature Interaction Network (HMRNet) for robust gait recognition. To facilitate LiDAR-based gait recognition research, we introduce FreeGait, a comprehensive gait dataset from large-scale, unconstrained settings.
  arXiv  Detail & Related papers  (2022-11-22T16:05:58Z)
• LidarGait: Benchmarking 3D Gait Recognition with Point Clouds [18.22238384814974]
  This work explores precise 3D gait features from point clouds and proposes a simple yet efficient 3D gait recognition framework, termed LidarGait. Our proposed approach projects sparse point clouds into depth maps to learn the representations with 3D geometry information. Due to the lack of point cloud datasets, we built the first large-scale LiDAR-based gait recognition dataset, SUSTech1K.
  arXiv  Detail & Related papers  (2022-11-19T06:23:08Z)
• Generative Range Imaging for Learning Scene Priors of 3D LiDAR Data [3.9447103367861542]
  This paper proposes a generative model of LiDAR range images applicable to the data-level domain transfer. Motivated by the fact that LiDAR measurement is based on point-by-point range imaging, we train an implicit image representation-based generative adversarial networks. We demonstrate the fidelity and diversity of our model in comparison with the point-based and image-based state-of-the-art generative models.
  arXiv  Detail & Related papers  (2022-10-21T06:08:39Z)
• MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detection [89.26380781863665]
  Fusing LiDAR and camera information is essential for achieving accurate and reliable 3D object detection in autonomous driving systems. Recent approaches aim at exploring the semantic densities of camera features through lifting points in 2D camera images into 3D space for fusion. We propose a novel framework that focuses on the multi-scale progressive interaction of the multi-granularity LiDAR and camera features.
  arXiv  Detail & Related papers  (2022-09-07T12:29:29Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• LIF-Seg: LiDAR and Camera Image Fusion for 3D LiDAR Semantic Segmentation [78.74202673902303]
  We propose a coarse-tofine LiDAR and camera fusion-based network (termed as LIF-Seg) for LiDAR segmentation. The proposed method fully utilizes the contextual information of images and introduces a simple but effective early-fusion strategy. The cooperation of these two components leads to the success of the effective camera-LiDAR fusion.
  arXiv  Detail & Related papers  (2021-08-17T08:53:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.