Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration

• URL: http://arxiv.org/abs/2608.02285v1
• Date: Mon, 03 Aug 2026 14:16:18 GMT
• Title: Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration
• Abstract summary: Sen-Cap is a 3D motion capture framework that integrates multi-modal data from LiDAR and camera.<n>Sen-Cap achieves state-of-the-art performance on major metrics on Human-M3 and FreeMotion.<n>This combination of flexibility and robustness opens new opportunities for motion capture in real-world scenarios.
• Score: 50.18488308004802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose Sen-Cap, a Sensor-Flexible and Noise-Resilient 3D human motion Capture framework that integrates multi-modal data from LiDAR and camera. While multi-modal sensors provide richer information than single-modal sensors, existing approaches still suffer from two core challenges. First, multi-modal alignment/matching across arbitrarily deployed sensors is typically handled by explicit calibration, which propagates errors under changing viewpoints and in turn constrains deployment to fixed, highly overlapped layouts. Second, prior methods degrade under severe noise or partial sensor failures, which are common in real-world environments. To address these challenges, Sen-Cap introduces a Unified Across-Sensor Motion Estimator that reconstructs local pose and shape in a human-centric space without calibrations between sensors, supporting a flexible number of sensors, as well as a Noise-Resistant Trajectory Tracker that maintains robustness under severe point cloud noise through iterative refinement. These sensor-flexible and noise-resilient features make Sen-Cap more practical in real-world deployment. Notably, operating in real time, Sen-Cap achieves state-of-the-art performance on major metrics on Human-M3 and FreeMotion, as well as strong cross-domain performance on LiDARHuman26M and RELI11D. This combination of flexibility and robustness opens new opportunities for motion capture in real-world scenarios, e.g. sports analytics, field robotics, and large-scale immersive environments.

Related papers

• Talk2Sensors: 3D Visual Grounding in Autonomous Driving via Sensor-Adaptive Physical Cue Matching [66.91228384649921]
  We introduce Talk2Sensors, the first multi-sensor 3D visual grounding dataset built upon camera, LiDAR, and 4D radar.<n>We propose TSFormer, a unified Transformer-based framework for language-guided 3D visual grounding in autonomous driving.
  arXiv  Detail & Related papers  (2026-08-05T07:58:35Z)
• Towards Real-World Wearable Motion Reconstruction [68.1355376294881]
  We present a large-scale dataset synchronizing consumer-grade sensors with ground-truth 3D motion.<n>Second, we propose WHIP, a baseline generative model that reconstructs motion from arbitrary subsets of available sensors.<n>Third, we conduct a systematic study of sensor complementarity, quantifying how different modalities complement one another.
  arXiv  Detail & Related papers  (2026-07-08T11:56:02Z)
• WeatherOcc3D: VLM-Assisted Adverse Weather Aware 3D Semantic Occupancy Prediction [0.0]
  We propose a VLM-assisted framework to guide multi-sensor integration via linguistic environmental cues.<n>We utilize a parameter-efficient adapter to align weather-specific text embeddings with sensor features, coupled with a gating strategy.<n>This enables the model to dynamically modulate the fusion ratio - prioritizing semantic camera features in clear daylight and shifting to geometric LiDAR priors during rainy nights.
  arXiv  Detail & Related papers  (2026-05-15T16:12:50Z)
• ReSPEC: A Framework for Online Multispectral Sensor Reconfiguration in Dynamic Environments [6.718801307135799]
  We introduce a framework that unifies sensing, learning, and actuation into a closed reconfiguration loop.<n>We implement and evaluate this framework on a mobile rover, showing that adaptive control reduces load by 29.3% with only a 5.3% accuracy drop compared to a baseline.
  arXiv  Detail & Related papers  (2026-02-11T05:39:24Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• Occluded nuScenes: A Multi-Sensor Dataset for Evaluating Perception Robustness in Automated Driving [4.60199848319746]
  Occluded nuScenes dataset is a novel extension of the widely used nuScenes benchmark.<n>This resource supports consistent, reproducible evaluation of perception models under partial sensor failures and environmental interference.<n>We aim to advance research on robust sensor fusion, resilience analysis, and safety-critical perception in automated driving.
  arXiv  Detail & Related papers  (2025-10-21T12:02:26Z)
• SMARTIES: Spectrum-Aware Multi-Sensor Auto-Encoder for Remote Sensing Images [17.304106221330414]
  Recent deep learning models are often specific to single sensors or to fixed combinations.<n>We introduce SMARTIES, a generic and versatile foundation model lifting sensor-specific/dependent efforts.<n>On both single and multi-modal tasks across diverse sensors, SMARTIES outperforms previous models that rely on sensor-specific pretraining.
  arXiv  Detail & Related papers  (2025-06-24T12:51:39Z)
• Progressive Inertial Poser: Progressive Real-Time Kinematic Chain Estimation for 3D Full-Body Pose from Three IMU Sensors [25.67875816218477]
  Full-body pose estimation from sparse tracking signals is not limited by environmental conditions or recording range.<n>Previous works either face the challenge of wearing additional sensors on the pelvis and lower-body or rely on external visual sensors to obtain global positions of key joints.<n>To improve the practicality of the technology for virtual reality applications, we estimate full-body poses using only inertial data obtained from three Inertial Measurement Unit (IMU) sensors worn on the head and wrists.
  arXiv  Detail & Related papers  (2025-05-08T15:28:09Z)
• MSSIDD: A Benchmark for Multi-Sensor Denoising [55.41612200877861]
  We introduce a new benchmark, the Multi-Sensor SIDD dataset, which is the first raw-domain dataset designed to evaluate the sensor transferability of denoising models. We propose a sensor consistency training framework that enables denoising models to learn the sensor-invariant features.
  arXiv  Detail & Related papers  (2024-11-18T13:32:59Z)
• CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes [56.52618054240197]
  We propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes.<n>Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token.<n>Our model significantly improves robustness and accuracy, especially in adverse-condition scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.