DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion

• URL: http://arxiv.org/abs/2308.16682v2
• Date: Thu, 28 Mar 2024 15:49:42 GMT
• Title: DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion
• Authors: Tom Van Wouwe, Seunghwan Lee, Antoine Falisse, Scott Delp, C. Karen Liu,
• Abstract summary: Motion capture from a limited number of body-worn sensors has important applications in health, human performance, and entertainment. Recent work has focused on accurately reconstructing whole-body motion from a specific sensor configuration using six IMUs. We propose a single diffusion model, DiffusionPoser, which reconstructs human motion in real-time from an arbitrary combination of sensors.
• Score: 10.439802168557513
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Motion capture from a limited number of body-worn sensors, such as inertial measurement units (IMUs) and pressure insoles, has important applications in health, human performance, and entertainment. Recent work has focused on accurately reconstructing whole-body motion from a specific sensor configuration using six IMUs. While a common goal across applications is to use the minimal number of sensors to achieve required accuracy, the optimal arrangement of the sensors might differ from application to application. We propose a single diffusion model, DiffusionPoser, which reconstructs human motion in real-time from an arbitrary combination of sensors, including IMUs placed at specified locations, and, pressure insoles. Unlike existing methods, our model grants users the flexibility to determine the number and arrangement of sensors tailored to the specific activity of interest, without the need for retraining. A novel autoregressive inferencing scheme ensures real-time motion reconstruction that closely aligns with measured sensor signals. The generative nature of DiffusionPoser ensures realistic behavior, even for degrees-of-freedom not directly measured. Qualitative results can be found on our website: https://diffusionposer.github.io/.

Related papers

• Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes [56.52618054240197]
  We propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes. Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token that guides the fusion of multiple sensor modalities. We set the new state of the art with CAFuser on the MUSES dataset with 59.7 PQ for multimodal panoptic segmentation and 78.2 mIoU for semantic segmentation, ranking first on the public benchmarks.
  arXiv  Detail & Related papers  (2024-10-14T17:56:20Z)
• Spatial-Related Sensors Matters: 3D Human Motion Reconstruction Assisted with Textual Semantics [4.9493039356268875]
  Leveraging wearable devices for motion reconstruction has emerged as an economical and viable technique. In this paper, we explore the spatial importance of multiple sensors, supervised by text that describes specific actions. With textual supervision, our method not only differentiates between ambiguous actions such as sitting and standing but also produces more precise and natural motion.
  arXiv  Detail & Related papers  (2023-12-27T04:21:45Z)
• Physical-Layer Semantic-Aware Network for Zero-Shot Wireless Sensing [74.12670841657038]
  Device-free wireless sensing has recently attracted significant interest due to its potential to support a wide range of immersive human-machine interactive applications. Data heterogeneity in wireless signals and data privacy regulation of distributed sensing have been considered as the major challenges that hinder the wide applications of wireless sensing in large area networking systems. We propose a novel zero-shot wireless sensing solution that allows models constructed in one or a limited number of locations to be directly transferred to other locations without any labeled data.
  arXiv  Detail & Related papers  (2023-12-08T13:50:30Z)
• Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognition [5.225544155289783]
  Various types of sensors can be used for Human Activity Recognition (HAR) Sometimes a single sensor cannot fully observe the user's motions from its perspective, which causes wrong predictions. We propose Virtual Fusion - a new method that takes advantage of unlabeled data from multiple time-synchronized sensors during training, but only needs one sensor for inference.
  arXiv  Detail & Related papers  (2023-12-01T17:03:27Z)
• Design Space Exploration on Efficient and Accurate Human Pose Estimation from Sparse IMU-Sensing [0.04594153909580514]
  Human Pose Estimation (HPE) to assess human motion in sports, rehabilitation or work safety requires accurate sensing without compromising personal data. Central trade-off between accuracy and efficient use of hardware resources is rarely discussed in research. We generate IMU-data from a publicly available body model dataset for different sensor configurations and train a deep learning model with this data.
  arXiv  Detail & Related papers  (2023-07-21T13:36:49Z)
• Environmental Sensor Placement with Convolutional Gaussian Neural Processes [65.13973319334625]
  It is challenging to place sensors in a way that maximises the informativeness of their measurements, particularly in remote regions like Antarctica. Probabilistic machine learning models can suggest informative sensor placements by finding sites that maximally reduce prediction uncertainty. This paper proposes using a convolutional Gaussian neural process (ConvGNP) to address these issues.
  arXiv  Detail & Related papers  (2022-11-18T17:25:14Z)
• Attention-Based Sensor Fusion for Human Activity Recognition Using IMU Signals [4.558966602878624]
  We propose a novel attention-based approach to human activity recognition using multiple IMU sensors worn at different body locations. An attention-based fusion mechanism is developed to learn the importance of sensors at different body locations and to generate an attentive feature representation. The proposed approach is evaluated using five public datasets and it outperforms state-of-the-art methods on a wide variety of activity categories.
  arXiv  Detail & Related papers  (2021-12-20T17:00:27Z)
• Motion Detection using CSI from Raspberry Pi 4 [5.826796031213696]
  Channel State Information (CSI) is a low cost, unintrusive form of radio sensing. We have developed a novel, self-calibrating motion detection system which uses CSI data collected and processed on a stock Raspberry Pi 4.
  arXiv  Detail & Related papers  (2021-11-17T13:17:02Z)
• Bandit Quickest Changepoint Detection [55.855465482260165]
  Continuous monitoring of every sensor can be expensive due to resource constraints. We derive an information-theoretic lower bound on the detection delay for a general class of finitely parameterized probability distributions. We propose a computationally efficient online sensing scheme, which seamlessly balances the need for exploration of different sensing options with exploitation of querying informative actions.
  arXiv  Detail & Related papers  (2021-07-22T07:25:35Z)
• Real-time detection of uncalibrated sensors using Neural Networks [62.997667081978825]
  An online machine-learning based uncalibration detector for temperature, humidity and pressure sensors was developed. The solution integrates an Artificial Neural Network as main component which learns from the behavior of the sensors under calibrated conditions. The obtained results show that the proposed solution is able to detect uncalibrations for deviation values of 0.25 degrees, 1% RH and 1.5 Pa, respectively.
  arXiv  Detail & Related papers  (2021-02-02T15:44:39Z)
• Deep Soft Procrustes for Markerless Volumetric Sensor Alignment [81.13055566952221]
  In this work, we improve markerless data-driven correspondence estimation to achieve more robust multi-sensor spatial alignment. We incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one. Our model is experimentally shown to achieve similar results with marker-based methods and outperform the markerless ones, while also being robust to the pose variations of the calibration structure.
  arXiv  Detail & Related papers  (2020-03-23T10:51:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.