Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G

• URL: http://arxiv.org/abs/2606.23771v1
• Date: Mon, 22 Jun 2026 17:27:42 GMT
• Title: Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G
• Abstract summary: Extended Reality (XR) presents a challenging use case for 5G and 6G networks.<n>Current XR interaction solutions based on handheld controllers and cameras cannot easily capture full-body poses.<n>We propose a multimodal sensing architecture for XR that combines 5G MillimeterWave (mmWave) Integrated sensing and communication (ISAC) and surface electromyography (sEMG) signals.
• Score: 3.8468982193073757
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Extended Reality (XR) presents a challenging use case for 5G and 6G networks, requiring high data-rates and lowlatency communication to deliver a truly immersive experience. Moreover, in order to seamlessly translate physical actions to the virtual world, accurate gesture recognition and pose estimation are required. Current XR interaction solutions based on handheld controllers and cameras cannot easily capture full-body poses, inhibit the free use of hands, and require good visibility and a clear line of sight. In this work, we propose a multimodal sensing architecture for XR that combines 5G MillimeterWave (mmWave) Integrated sensing and communication (ISAC) and surface electromyography (sEMG) signals. 5G mmWave ISAC cannot only be used to deliver content wirelessly to the Head-mounted display (HMD), but also the same communication signals can be used to derive coarse body-level gestures and poses of the user, to support real-time avatar control. For fine-grained finger-level gestures, our architecture leverages lightweight sEMG sensors that capture forearm muscle activity. To illustrate the need of both modalities, we present evaluations of both sensing technologies. At the body level (5G), our architecture relies on power-per-beam-pair (PPBP), which can be computed from standard beam management or beam sweeping procedures of the 5G NR standard. PPBP-based sensing achieves 82.2$\pm$5.9% average accuracy when evaluated on users not seen during training. For fine-grained finger-level interactions, we show that surface electromyography (sEMG) carries strong discriminative information achieving consistent promising performance across different movement settings. Thus, combining the two modalities enables multi-scale gesture recognition, at the body level via existing 5G signals and finger level via lightweight sEMG sensors, forming a complete XR framework.

Related papers

• Talk2Sensors: 3D Visual Grounding in Autonomous Driving via Sensor-Adaptive Physical Cue Matching [66.91228384649921]
  We introduce Talk2Sensors, the first multi-sensor 3D visual grounding dataset built upon camera, LiDAR, and 4D radar.<n>We propose TSFormer, a unified Transformer-based framework for language-guided 3D visual grounding in autonomous driving.
  arXiv  Detail & Related papers  (2026-08-05T07:58:35Z)
• Sen-Cap: Sensor-Flexible and Noise-Resilient Human Motion Capture via LiDAR-Camera Integration [50.18488308004802]
  Sen-Cap is a 3D motion capture framework that integrates multi-modal data from LiDAR and camera.<n>Sen-Cap achieves state-of-the-art performance on major metrics on Human-M3 and FreeMotion.<n>This combination of flexibility and robustness opens new opportunities for motion capture in real-world scenarios.
  arXiv  Detail & Related papers  (2026-08-03T14:16:18Z)
• Real-Time sEMG-Based Telecontrol of an Assistive Robotic Arm Using a 1D Convolutional Neural Network [0.30786914102688595]
  This work proposes a real-time sEMG-based interface for the teleoperation of an assistive robotic arm.<n>The system relies on four-channel sEMG acquisition, signal preprocessing, segmentation into sliding windows, and classification using a one-dimensional convolutional neural network (CNN)<n>The system exhibits stable real-time behavior, with an average latency of approximately 0.32 s consistent with the chosen windowing strategy.
  arXiv  Detail & Related papers  (2026-07-14T15:34:28Z)
• RAWild: Sensor-Agnostic RAW Object Detection via Physics-Guided Curve and Grid Modeling [69.96407948879015]
  textbfRAWild is a physics-guided global-local tone mapping framework for sensor-agnostic RAW object detection.<n>We construct a physics-based RAW simulation pipeline that synthesizes realistic sensor outputs spanning diverse spectral sensitivities, illuminants, and sensor non-idealities.
  arXiv  Detail & Related papers  (2026-05-07T09:51:47Z)
• DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization [5.972698171130012]
  High-fidelity teleoperation of dexterous robotic hands is essential for bringing robots into unstructured domestic environments.<n>We develop a lightweight and cost-effective teleoperation system leveraging surface electromyography (sEMG)<n>We show that our system achieves high precision in diverse teleoperation tasks.
  arXiv  Detail & Related papers  (2026-03-06T03:36:33Z)
• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion [14.332919759770645]
  Vision-based gesture recognition has been explored as one method for hands-free teleoperation.<n>We propose a multimodal gesture recognition framework that integrates inertial data from Apple Watches on both wrists with capacitive sensing signals from custom gloves.<n>We show that our framework achieves performance comparable to a state-of-the-art vision-based baseline.
  arXiv  Detail & Related papers  (2026-02-27T05:52:04Z)
• UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors [51.88112610411651]
  We propose a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors.<n>UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image)<n>To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions.
  arXiv  Detail & Related papers  (2026-02-01T11:03:01Z)
• emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation [12.566524562446467]
  Reliable and always-available hand pose inference could yield new and intuitive control schemes for human-computer interactions.<n>Wearable wrist-based surface electromyography (sEMG) presents a promising alternative.<n>emg2pose is the largest publicly available dataset of high-quality hand pose labels and wrist sEMG recordings.
  arXiv  Detail & Related papers  (2024-12-02T23:39:37Z)
• Agile gesture recognition for capacitive sensing devices: adapting on-the-job [55.40855017016652]
  We demonstrate a hand gesture recognition system that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer five fingers. We use a machine learning technique to analyse the time series signals and identify three features that can represent 5 fingers within 500 ms.
  arXiv  Detail & Related papers  (2023-05-12T17:24:02Z)
• DensePose From WiFi [86.61881052177228]
  We develop a deep neural network that maps the phase and amplitude of WiFi signals to UV coordinates within 24 human regions. Our model can estimate the dense pose of multiple subjects, with comparable performance to image-based approaches.
  arXiv  Detail & Related papers  (2022-12-31T16:48:43Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.