Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs

• URL: http://arxiv.org/abs/2604.21926v1
• Date: Thu, 23 Apr 2026 17:59:16 GMT
• Title: Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs
• Abstract summary: We introduce IMU-to--4D, a framework that reconstructs human motion and 3D layouts purely from wearable sensors.<n>IMU-to--4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with coarse scene structure.<n>Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than SoTAd pipelines.
• Score: 85.01118931556098
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding human activities and their surrounding environments typically relies on visual perception, yet cameras pose persistent challenges in privacy, safety, energy efficiency, and scalability. We explore an alternative: 4D perception without vision. Its goal is to reconstruct human motion and 3D scene layouts purely from everyday wearable sensors. For this we introduce IMU-to-4D, a framework that repurposes large language models for non-visual spatiotemporal understanding of human-scene dynamics. IMU-to-4D uses data from a few inertial sensors from earbuds, watches, or smartphones and predicts detailed 4D human motion together with coarse scene structure. Experiments across diverse human-scene datasets show that IMU-to-4D yields more coherent and temporally stable results than SoTA cascaded pipelines, suggesting wearable motion sensors alone can support rich 4D understanding.

Related papers

• Syn4D: A Multiview Synthetic 4D Dataset [92.50132053229817]
  3D reconstruction and tracking of dynamic scenes from monocular video remains an important open challenge in computer vision.<n>To address this limitation, we introduce Syn4D, a multiview synthetic dynamic scenes.<n>Key feature of Syn4D is the ability to unproject any pixel into 3D.
  arXiv  Detail & Related papers  (2026-05-06T17:59:55Z)
• EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents [85.77432303199176]
  We propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones.<n>Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes.<n>Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via
  arXiv  Detail & Related papers  (2026-02-26T16:53:41Z)
• 4D-based Robot Navigation Using Relativistic Image Processing [0.0]
  4D perception offers the possibility of predicting one's own position and changes in the environment over time. We present a 4D-based approach to robot navigation using relativistic image processing.
  arXiv  Detail & Related papers  (2024-10-29T14:42:19Z)
• HiSC4D: Human-centered interaction and 4D Scene Capture in Large-scale Space Using Wearable IMUs and LiDAR [43.43745311617461]
  We introduce HiSC4D, a novel Human-centered interaction and 4D Scene Capture method. By utilizing body-mounted IMUs and a head-mounted LiDAR, HiSC4D can capture egocentric human motions in unconstrained space. We present a dataset, containing 8 sequences in 4 large scenes (200 to 5,000 $m2$), providing 36k frames of accurate 4D human motions.
  arXiv  Detail & Related papers  (2024-09-06T16:43:04Z)
• TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments [106.80978555346958]
  Current methods can't reliably estimate moving humans in global coordinates. TRACE is the first one-stage method to jointly recover and track 3D humans in global coordinates from dynamic cameras. It achieves state-of-the-art performance on tracking and HPS benchmarks.
  arXiv  Detail & Related papers  (2023-06-05T13:00:44Z)
• HSC4D: Human-centered 4D Scene Capture in Large-scale Indoor-outdoor Space Using Wearable IMUs and LiDAR [51.9200422793806]
  Using only body-mounted IMUs and LiDAR, HSC4D is space-free without any external devices' constraints and map-free without pre-built maps. Relationships between humans and environments are also explored to make their interaction more realistic.
  arXiv  Detail & Related papers  (2022-03-17T10:05:55Z)
• Learning Motion Priors for 4D Human Body Capture in 3D Scenes [81.54377747405812]
  We propose LEMO: LEarning human MOtion priors for 4D human body capture. We introduce a novel motion prior, which reduces the jitters exhibited by poses recovered over a sequence. We also design a contact friction term and a contact-aware motion infiller obtained via per-instance self-supervised training. With our pipeline, we demonstrate high-quality 4D human body capture, reconstructing smooth motions and physically plausible body-scene interactions.
  arXiv  Detail & Related papers  (2021-08-23T20:47:09Z)
• Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors [71.29186299435423]
  We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment. We show that our optimization-based integration exploits the benefits of the two, resulting in pose accuracy free of drift. HPS could be used for VR/AR applications where humans interact with the scene without requiring direct line of sight with an external camera.
  arXiv  Detail & Related papers  (2021-03-31T17:58:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.