EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation

• URL: http://arxiv.org/abs/2605.05712v1
• Date: Thu, 07 May 2026 05:56:57 GMT
• Title: EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation
• Abstract summary: EgoEMG is a multimodal egocentric dataset for bimanual hand pose estimation.<n>The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures.<n>EgoEMG and its benchmark establish a foundation for future research on multimodal hand pose estimation with EMG and vision.
• Score: 21.77386351667022
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Surface electromyography (sEMG) records muscle activity during hand movement and can be decoded to recover detailed hand articulation. EMG and egocentric vision are complementary for hand sensing: EMG captures fine-grained finger articulation even under occlusion and poor lighting, while vision provides global hand configuration. However, no existing dataset synchronizes both modalities. We present EgoEMG, a multimodal egocentric dataset for bimanual hand pose estimation. EgoEMG includes bilateral wristband EMG with 16 total channels (8 per wrist) sampled at 2 kHz, 120 Hz IMU, egocentric wide-angle RGB video, external RGB-D video, and mocap-derived hand motion with wrist articulation angles. The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures, totaling more than 10 hours of recording. We also introduce a benchmark with three tasks -- EMG-to-pose, vision-to-pose, and EMG+vision fusion -- under a shared joint-angle prediction target and common generalization split axes (cross-gesture, cross-user, and combined). As baselines, we evaluate EMGFormer for EMG-to-pose and generic ResNet/ViT backbones for vision-to-pose. We further study a residual fusion architecture that improves over matched lightweight vision-only baselines. Together, EgoEMG and its benchmark establish a foundation for future research on multimodal hand pose estimation with EMG and vision.

Related papers

• DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery [51.92897182932217]
  Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
  arXiv  Detail & Related papers  (2026-08-20T17:46:24Z)
• HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control [51.913695868638634]
  We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video.<n>We build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes.<n>We also propose the Plcker Hand Map, a 3D-aware control signal that extends Plcker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level.
  arXiv  Detail & Related papers  (2026-07-02T12:14:02Z)
• GEM: Generative Supervision Helps Embodied Intelligence [60.28190143133428]
  Embodied Vision-Language Models (VLMs) have demonstrated impressive performance and generalization in robotics.<n>A significant gap remains between the high-level semantic focus of standard text-guided pre-training paradigms and the low-level spatial and physical knowledge critical for execution in embodied environments.<n>We introduce GEM, a Generative-supervised Embodied vision-language Model designed to bridge this divide.
  arXiv  Detail & Related papers  (2026-05-27T14:39:42Z)
• Decoding High-Dimensional Finger Motion from EMG Using Riemannian Features and RNNs [1.1534313664323634]
  We present an end-to-end framework for continuous EMG-to-kinematics regression using only consumer-grade hardware.<n>The framework combines an 8-channel EMG armband, a single webcam, and an automatic synchronization procedure.
  arXiv  Detail & Related papers  (2026-04-24T12:27:33Z)
• emg2tendon: From sEMG Signals to Tendon Control in Musculoskeletal Hands [5.613626927694011]
  Tendon-driven robotic hands offer unparalleled dexterity for manipulation tasks.<n>However, learning control policies for such systems presents unique challenges.<n>We introduce the first large-scale EMG-to-Tendon Control dataset for robotic hands.
  arXiv  Detail & Related papers  (2025-07-29T12:49:57Z)
• EgoM2P: Egocentric Multimodal Multitask Pretraining [55.259234688003545]
  Building large-scale egocentric multimodal and multitask models presents unique challenges.<n> EgoM2P is a masked modeling framework that learns from temporally-aware multimodal tokens to train a large, general-purpose model for egocentric 4D understanding.<n>We will fully open-source EgoM2P to support the community and advance egocentric vision research.
  arXiv  Detail & Related papers  (2025-06-09T15:59:25Z)
• EgoExOR: An Ego-Exo-Centric Operating Room Dataset for Surgical Activity Understanding [43.66860935790616]
  EgoExOR is the first operating room (OR) dataset to fuse first-person and third-person perspectives.<n>It integrates egocentric data (RGB, gaze, hand tracking, audio) from wearable glasses, exocentric RGB and depth from RGB-D cameras, and ultrasound imagery.<n>We evaluate the surgical scene graph generation performance of two adapted state-of-the-art models.
  arXiv  Detail & Related papers  (2025-05-30T07:02:00Z)
• GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Images [44.50428701650495]
  We introduce GEM, the first MLLM unifying ECG time series, 12-lead ECG images and text for grounded and clinician-aligned ECG interpretation.<n> GEM enables feature-grounded analysis, evidence-driven reasoning, and a clinician-like diagnostic process through three core innovations.<n>We propose the Grounded ECG task, a clinically motivated benchmark designed to assess the MLLM's capability in grounded ECG understanding.
  arXiv  Detail & Related papers  (2025-03-08T05:48:53Z)
• GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control [122.65089441381741]
  We present GEM, a Generalizable Ego-vision Multimodal world model.<n>It predicts future frames using a reference frame, sparse features, human poses, and ego-trajectories.<n>Our dataset is comprised of 4000+ hours of multimodal data across domains like autonomous driving, egocentric human activities, and drone flights.
  arXiv  Detail & Related papers  (2024-12-15T14:21:19Z)
• emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation [12.566524562446467]
  Reliable and always-available hand pose inference could yield new and intuitive control schemes for human-computer interactions.<n>Wearable wrist-based surface electromyography (sEMG) presents a promising alternative.<n>emg2pose is the largest publicly available dataset of high-quality hand pose labels and wrist sEMG recordings.
  arXiv  Detail & Related papers  (2024-12-02T23:39:37Z)
• SiMA-Hand: Boosting 3D Hand-Mesh Reconstruction by Single-to-Multi-View Adaptation [90.59734612754222]
  Estimating 3D hand mesh from RGB images is one of the most challenging problems. Existing attempts towards this task often fail when the occlusion dominates the image space. We propose SiMA-Hand, aiming to boost the mesh reconstruction performance by Single-to-Multi-view Adaptation.
  arXiv  Detail & Related papers  (2024-02-02T13:14:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.