mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstruction

• URL: http://arxiv.org/abs/2503.02375v1
• Date: Tue, 04 Mar 2025 08:03:53 GMT
• Title: mmDEAR: mmWave Point Cloud Density Enhancement for Accurate Human Body Reconstruction
• Authors: Jiarui Yang, Songpengcheng Xia, Zengyuan Lai, Lan Sun, Qi Wu, Wenxian Yu, Ling Pei,
• Abstract summary: We propose a two-stage deep learning framework that enhances mmWave point clouds and improves body reconstruction accuracy.<n>Our approach outperforms state-of-the-art methods, with the enhanced point clouds further improving performance when integrated into existing models.
• Score: 14.480271406960467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Millimeter-wave (mmWave) radar offers robust sensing capabilities in diverse environments, making it a highly promising solution for human body reconstruction due to its privacy-friendly and non-intrusive nature. However, the significant sparsity of mmWave point clouds limits the estimation accuracy. To overcome this challenge, we propose a two-stage deep learning framework that enhances mmWave point clouds and improves human body reconstruction accuracy. Our method includes a mmWave point cloud enhancement module that densifies the raw data by leveraging temporal features and a multi-stage completion network, followed by a 2D-3D fusion module that extracts both 2D and 3D motion features to refine SMPL parameters. The mmWave point cloud enhancement module learns the detailed shape and posture information from 2D human masks in single-view images. However, image-based supervision is involved only during the training phase, and the inference relies solely on sparse point clouds to maintain privacy. Experiments on multiple datasets demonstrate that our approach outperforms state-of-the-art methods, with the enhanced point clouds further improving performance when integrated into existing models.

Related papers

• 3D Point Cloud Generation via Autoregressive Up-sampling [60.05226063558296]
  We introduce a pioneering autoregressive generative model for 3D point cloud generation. Inspired by visual autoregressive modeling, we conceptualize point cloud generation as an autoregressive up-sampling process. PointARU progressively refines 3D point clouds from coarse to fine scales.
  arXiv  Detail & Related papers  (2025-03-11T16:30:45Z)
• Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation [27.25933965875881]
  LiDAR-based 3D Human Pose Estimation is becoming a research focus.<n>Most of the existing methods use temporal information, multi-modal fusion, or SMPL optimization to correct biased results.<n>We propose a simple yet powerful method, which provides insights both on modeling and augmentation of point clouds.
  arXiv  Detail & Related papers  (2024-12-18T02:54:30Z)
• Towards Weather-Robust 3D Human Body Reconstruction: Millimeter-Wave Radar-Based Dataset, Benchmark, and Multi-Modal Fusion [13.082760040398147]
  3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather.<n>mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather.<n>We design ImmFusion, the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions.
  arXiv  Detail & Related papers  (2024-09-07T15:06:30Z)
• Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models [97.58685709663287]
  generative pre-training can boost the performance of fundamental models in 2D vision. In 3D vision, the over-reliance on Transformer-based backbones and the unordered nature of point clouds have restricted the further development of generative pre-training. We propose a novel 3D-to-2D generative pre-training method that is adaptable to any point cloud model.
  arXiv  Detail & Related papers  (2023-07-27T16:07:03Z)
• MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment [32.495387943305204]
  We propose a novel no-reference point cloud quality assessment (NR-PCQA) metric in a multi-modal fashion. In specific, we split the point clouds into sub-models to represent local geometry distortions such as point shift and down-sampling. To achieve the goals, the sub-models and projected images are encoded with point-based and image-based neural networks.
  arXiv  Detail & Related papers  (2022-09-01T06:11:12Z)
• Bridging the View Disparity of Radar and Camera Features for Multi-modal Fusion 3D Object Detection [6.959556180268547]
  This paper focuses on how to utilize millimeter-wave (MMW) radar and camera sensor fusion for 3D object detection. A novel method which realizes the feature-level fusion under bird-eye view (BEV) for a better feature representation is proposed.
  arXiv  Detail & Related papers  (2022-08-25T13:21:37Z)
• R2P: A Deep Learning Model from mmWave Radar to Point Cloud [14.803119281557995]
  Radar to Point Cloud (R2P) is a deep learning model that generates smooth, dense, and highly accurate point cloud representation of a 3D object. R2P replaces Stage 2 of our recently proposed 3DRIMR (3D Reconstruction and Imaging via mmWave Radar) system.
  arXiv  Detail & Related papers  (2022-07-21T18:01:05Z)
• DeepPoint: A Deep Learning Model for 3D Reconstruction in Point Clouds via mmWave Radar [10.119506666546663]
  We introduce in this paper DeepPoint, a deep learning model that generates 3D objects in point cloud format. It takes as input the 2D depth images of an object generated by 3DRIMR's Stage 1, and outputs smooth and dense 3D point clouds of the object. Our experiments have demonstrated that this model significantly outperforms the original 3DRIMR and other standard techniques in reconstructing 3D objects.
  arXiv  Detail & Related papers  (2021-09-19T18:28:20Z)
• EPMF: Efficient Perception-aware Multi-sensor Fusion for 3D Semantic Segmentation [62.210091681352914]
  We study multi-sensor fusion for 3D semantic segmentation for many applications, such as autonomous driving and robotics. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) We propose a two-stream network to extract features from the two modalities separately. The extracted features are fused by effective residual-based fusion modules.
  arXiv  Detail & Related papers  (2021-06-21T10:47:26Z)
• Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation [81.08111209632501]
  We propose a geometry-aware stereo-LiDAR fusion network for long-range depth estimation. We exploit sparse and accurate point clouds as a cue for guiding correspondences of stereo images in a unified 3D volume space. Our network achieves state-of-the-art performance on the KITTI and the Virtual- KITTI datasets.
  arXiv  Detail & Related papers  (2021-03-24T03:24:46Z)
• Cross-Modality 3D Object Detection [63.29935886648709]
  We present a novel two-stage multi-modal fusion network for 3D object detection. The whole architecture facilitates two-stage fusion. Our experiments on the KITTI dataset show that the proposed multi-stage fusion helps the network to learn better representations.
  arXiv  Detail & Related papers  (2020-08-16T11:01:20Z)
• Pseudo-LiDAR Point Cloud Interpolation Based on 3D Motion Representation and Spatial Supervision [68.35777836993212]
  We propose a Pseudo-LiDAR point cloud network to generate temporally and spatially high-quality point cloud sequences. By exploiting the scene flow between point clouds, the proposed network is able to learn a more accurate representation of the 3D spatial motion relationship.
  arXiv  Detail & Related papers  (2020-06-20T03:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.