LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR
Point Clouds
- URL: http://arxiv.org/abs/2203.14698v1
- Date: Mon, 28 Mar 2022 12:52:45 GMT
- Title: LiDARCap: Long-range Marker-less 3D Human Motion Capture with LiDAR
Point Clouds
- Authors: Jialian Li, Jingyi Zhang, Zhiyong Wang, Siqi Shen, Chenglu Wen, Yuexin
Ma, Lan Xu, Jingyi Yu, Cheng Wang
- Abstract summary: Existing motion capture datasets are largely short-range and cannot yet fit the need of long-range applications.
We propose LiDARHuman26M, a new human motion capture dataset captured by LiDAR at a much longer range to overcome this limitation.
Our dataset also includes the ground truth human motions acquired by the IMU system and the synchronous RGB images.
- Score: 58.402752909624716
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Existing motion capture datasets are largely short-range and cannot yet fit
the need of long-range applications. We propose LiDARHuman26M, a new human
motion capture dataset captured by LiDAR at a much longer range to overcome
this limitation. Our dataset also includes the ground truth human motions
acquired by the IMU system and the synchronous RGB images. We further present a
strong baseline method, LiDARCap, for LiDAR point cloud human motion capture.
Specifically, we first utilize PointNet++ to encode features of points and then
employ the inverse kinematics solver and SMPL optimizer to regress the pose
through aggregating the temporally encoded features hierarchically.
Quantitative and qualitative experiments show that our method outperforms the
techniques based only on RGB images. Ablation experiments demonstrate that our
dataset is challenging and worthy of further research. Finally, the experiments
on the KITTI Dataset and the Waymo Open Dataset show that our method can be
generalized to different LiDAR sensor settings.
Related papers
- Just Add $100 More: Augmenting NeRF-based Pseudo-LiDAR Point Cloud for Resolving Class-imbalance Problem [12.26293873825084]
We propose to leverage pseudo-LiDAR point clouds generated from videos capturing a surround view of miniatures or real-world objects of minor classes.
Our method, called Pseudo Ground Truth Augmentation (PGT-Aug), consists of three main steps: (i) volumetric 3D instance reconstruction using a 2D-to-3D view synthesis model, (ii) object-level domain alignment with LiDAR intensity estimation, and (iii) a hybrid context-aware placement method from ground and map information.
arXiv Detail & Related papers (2024-03-18T08:50:04Z) - LiDAR-based Person Re-identification [29.694346498355443]
We propose a LiDAR-based ReID framework, ReID3D, that utilizes pre-training strategy to retrieve features of 3D body shape.
To the best of our knowledge, we are the first to propose a solution for LiDAR-based ReID.
arXiv Detail & Related papers (2023-12-05T12:44:17Z) - LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields [112.62936571539232]
We introduce a new task, novel view synthesis for LiDAR sensors.
Traditional model-based LiDAR simulators with style-transfer neural networks can be applied to render novel views.
We use a neural radiance field (NeRF) to facilitate the joint learning of geometry and the attributes of 3D points.
arXiv Detail & Related papers (2023-04-20T15:44:37Z) - Learning to Simulate Realistic LiDARs [66.7519667383175]
We introduce a pipeline for data-driven simulation of a realistic LiDAR sensor.
We show that our model can learn to encode realistic effects such as dropped points on transparent surfaces.
We use our technique to learn models of two distinct LiDAR sensors and use them to improve simulated LiDAR data accordingly.
arXiv Detail & Related papers (2022-09-22T13:12:54Z) - Efficient Spatial-Temporal Information Fusion for LiDAR-Based 3D Moving
Object Segmentation [23.666607237164186]
We propose a novel deep neural network exploiting both spatial-temporal information and different representation modalities of LiDAR scans to improve LiDAR-MOS performance.
Specifically, we first use a range image-based dual-branch structure to separately deal with spatial and temporal information.
We also use a point refinement module via 3D sparse convolution to fuse the information from both LiDAR range image and point cloud representations.
arXiv Detail & Related papers (2022-07-05T17:59:17Z) - Boosting 3D Object Detection by Simulating Multimodality on Point Clouds [51.87740119160152]
This paper presents a new approach to boost a single-modality (LiDAR) 3D object detector by teaching it to simulate features and responses that follow a multi-modality (LiDAR-image) detector.
The approach needs LiDAR-image data only when training the single-modality detector, and once well-trained, it only needs LiDAR data at inference.
Experimental results on the nuScenes dataset show that our approach outperforms all SOTA LiDAR-only 3D detectors.
arXiv Detail & Related papers (2022-06-30T01:44:30Z) - LiDAR-aid Inertial Poser: Large-scale Human Motion Capture by Sparse
Inertial and LiDAR Sensors [38.60837840737258]
We propose a multi-sensor fusion method for capturing 3D human motions with accurate consecutive local poses and global trajectories in large-scale scenarios.
We design a two-stage pose estimator in a coarse-to-fine manner, where point clouds provide the coarse body shape and IMU measurements optimize the local actions.
We collect a LiDAR-IMU multi-modal mocap dataset, LIPD, with diverse human actions in long-range scenarios.
arXiv Detail & Related papers (2022-05-30T20:15:11Z) - Learning Moving-Object Tracking with FMCW LiDAR [53.05551269151209]
We propose a learning-based moving-object tracking method utilizing our newly developed LiDAR sensor, Frequency Modulated Continuous Wave (FMCW) LiDAR.
Given the labels, we propose a contrastive learning framework, which pulls together the features from the same instance in embedding space and pushes apart the features from different instances to improve the tracking quality.
arXiv Detail & Related papers (2022-03-02T09:11:36Z) - Cirrus: A Long-range Bi-pattern LiDAR Dataset [35.87501129332217]
We introduce Cirrus, a new long-range bi-pattern LiDAR public dataset for autonomous driving tasks.
Our platform is equipped with a high-resolution video camera and a pair of LiDAR sensors with a 250-meter effective range.
In Cirrus, eight categories of objects are exhaustively annotated in the LiDAR point clouds for the entire effective range.
arXiv Detail & Related papers (2020-12-05T03:18:31Z) - SelfVoxeLO: Self-supervised LiDAR Odometry with Voxel-based Deep Neural
Networks [81.64530401885476]
We propose a self-supervised LiDAR odometry method, dubbed SelfVoxeLO, to tackle these two difficulties.
Specifically, we propose a 3D convolution network to process the raw LiDAR data directly, which extracts features that better encode the 3D geometric patterns.
We evaluate our method's performances on two large-scale datasets, i.e., KITTI and Apollo-SouthBay.
arXiv Detail & Related papers (2020-10-19T09:23:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.