Towards Real-World Wearable Motion Reconstruction

• URL: http://arxiv.org/abs/2607.09780v1
• Date: Wed, 08 Jul 2026 11:56:02 GMT
• Title: Towards Real-World Wearable Motion Reconstruction
• Abstract summary: We present a large-scale dataset synchronizing consumer-grade sensors with ground-truth 3D motion.<n>Second, we propose WHIP, a baseline generative model that reconstructs motion from arbitrary subsets of available sensors.<n>Third, we conduct a systematic study of sensor complementarity, quantifying how different modalities complement one another.
• Score: 68.1355376294881
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The modern-day surge in popularity of wearable devices poses a fundamentally unique motion capture problem: reconstructing full-body movement from any set of sensing hardware worn at a given moment. Yet, most research efforts assume fixed sensor configurations (e.g. IMU suits or HMD-centric rigs) and cannot generalize across them. In contrast, we argue that motion capture should prioritize unobtrusive and lightweight devices such as smartphones, smartwatches, smart glasses, and smart insoles, and study the interplay between them. To this end, we make three contributions. First, we present a large-scale multi-modal dataset synchronizing these consumer-grade sensors with ground-truth 3D motion, spanning 50 diverse activities including everyday tasks, sports, and social interactions. Second, we propose WHIP, a baseline generative model that reconstructs motion from arbitrary subsets of available sensors, robustly handling missing modalities and producing physically plausible motions. Third, we conduct a systematic study of sensor complementarity, quantifying how different modalities complement one another. Code and dataset are available at https://vcai.mpi-inf.mpg.de/projects/WHIP/

Related papers

• Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration [50.18488308004802]
  Sen-Cap is a 3D motion capture framework that integrates multi-modal data from LiDAR and camera.<n>Sen-Cap achieves state-of-the-art performance on major metrics on Human-M3 and FreeMotion.<n>This combination of flexibility and robustness opens new opportunities for motion capture in real-world scenarios.
  arXiv  Detail & Related papers  (2026-08-03T14:16:18Z)
• EgoExoMoCap: Distributed Ego-Exo Human Motion Capture [38.54314764814604]
  Human motion capture from head-mounted devices (HMDs) offers a scalable way to acquire real-world human motion and interaction data.<n>We propose a novel distributed framework that jointly leverages ego- and exocentric multi-modal signals for human motion estimation from HMDs.<n>Our approach, EgoExoMoCap, is as simple as two (or more) people, each wearing a pair of smart glasses.
  arXiv  Detail & Related papers  (2026-07-17T11:31:41Z)
• Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input [62.51283548975632]
  This work focuses on tracking and understanding human motion using consumer wearable devices, such as VR/AR headsets, smart glasses, cellphones, and smartwatches.<n>We present Ego4o (o for omni), a new framework for simultaneous human motion capture and understanding from multi-modal egocentric inputs.
  arXiv  Detail & Related papers  (2025-04-11T11:18:57Z)
• Suite-IN: Aggregating Motion Features from Apple Suite for Robust Inertial Navigation [10.634236058278722]
  Motion data captured by sensors on different body parts contains both local and global motion information. We propose a multi-device deep learning framework named Suite-IN, aggregating motion data from Apple Suite for inertial navigation.
  arXiv  Detail & Related papers  (2024-11-12T14:23:52Z)
• Sitcom-Crafter: A Plot-Driven Human Motion Generation System in 3D Scenes [90.39860012099393]
  Sitcom-Crafter is a system for human motion generation in 3D space.<n>Central to the function generation modules is our novel 3D scene-aware human-human interaction module.<n> Augmentation modules encompass plot comprehension for command generation, motion synchronization for seamless integration of different motion types.
  arXiv  Detail & Related papers  (2024-10-14T17:56:19Z)
• Motion Capture from Inertial and Vision Sensors [73.3003181330859]
  We contribute MINIONS, a large-scale Motion capture dataset collected from INertial and visION Sensors.<n>With the proposed MINIONS dataset, we propose a SparseNet framework to capture human motion from IMUs and videos.<n>Experiment results emphasize the unique advantages of inertial and vision sensors, showcasing the promise of consumer-affordable multi-modal motion capture.
  arXiv  Detail & Related papers  (2024-07-23T09:41:10Z)
• I'M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions [42.87514729260336]
  I'm-HOI is a monocular scheme to faithfully capture the 3D motions of both the human and object in a novel setting. It combines general motion inference and category-aware refinement. Our dataset and code will be released to the community.
  arXiv  Detail & Related papers  (2023-12-10T08:25:41Z)
• SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data [1.494051815405093]
  We introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from sparse data. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices.
  arXiv  Detail & Related papers  (2023-11-03T18:48:01Z)
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars [80.05743236282564]
  Real-time tracking of human body motion is crucial for immersive experiences in AR/VR. We present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers. We show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
  arXiv  Detail & Related papers  (2022-09-20T00:25:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.