emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation

• URL: http://arxiv.org/abs/2412.02725v1
• Date: Mon, 02 Dec 2024 23:39:37 GMT
• Title: emg2pose: A Large and Diverse Benchmark for Surface Electromyographic Hand Pose Estimation
• Abstract summary: Reliable and always-available hand pose inference could yield new and intuitive control schemes for human-computer interactions.<n>Wearable wrist-based surface electromyography (sEMG) presents a promising alternative.<n>emg2pose is the largest publicly available dataset of high-quality hand pose labels and wrist sEMG recordings.
• Score: 12.566524562446467
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Hands are the primary means through which humans interact with the world. Reliable and always-available hand pose inference could yield new and intuitive control schemes for human-computer interactions, particularly in virtual and augmented reality. Computer vision is effective but requires one or multiple cameras and can struggle with occlusions, limited field of view, and poor lighting. Wearable wrist-based surface electromyography (sEMG) presents a promising alternative as an always-available modality sensing muscle activities that drive hand motion. However, sEMG signals are strongly dependent on user anatomy and sensor placement, and existing sEMG models have required hundreds of users and device placements to effectively generalize. To facilitate progress on sEMG pose inference, we introduce the emg2pose benchmark, the largest publicly available dataset of high-quality hand pose labels and wrist sEMG recordings. emg2pose contains 2kHz, 16 channel sEMG and pose labels from a 26-camera motion capture rig for 193 users, 370 hours, and 29 stages with diverse gestures - a scale comparable to vision-based hand pose datasets. We provide competitive baselines and challenging tasks evaluating real-world generalization scenarios: held-out users, sensor placements, and stages. emg2pose provides the machine learning community a platform for exploring complex generalization problems, holding potential to significantly enhance the development of sEMG-based human-computer interactions.

Related papers

• HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object-Interaction [48.4618478154232]
  HiPHI is a 600+ hour scale high-fidelity whole-body human motion dataset.<n>It provides sub-millimeter spatial marker tracking accuracy for full-body human motion and mesh-level object trajectories.<n>We introduce a benchmark suite evaluating motion-space diversity, interaction grounding, object consistency, and physical AI applications.
  arXiv  Detail & Related papers  (2026-08-17T07:54:26Z)
• HandEdit: A Unified Benchmark for Egocentric Human-to-Robot Dexterous Hand Image Editing [96.6894553388557]
  HandEdit is a large-scale embodiment-aware image-editing dataset and benchmark.<n>It transforms human hands and arms into various dexterous robotic embodiments within egocentric frames.<n>HandEdit comprises over 200M editing instances derived from five diverse source datasets.
  arXiv  Detail & Related papers  (2026-08-12T14:41:53Z)
• ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine [83.02804728003862]
  Ambient Capture Engine (ACE) transforms real home environments into spatially, temporally synchronized recording studios.<n>ACE records egocentric and multi-view exocentric video, full-body and articulated hand motion, object geometry and 6-DoF trajectories, audio, and signals as a unified multisensory stream.<n>Dataset spans atomic manipulation, long-horizon chains of household activities, and human-scene interaction, while preserving natural behavioral variation through goal-level rather than step-by-step instructions.
  arXiv  Detail & Related papers  (2026-07-30T17:59:50Z)
• Towards Real-World Wearable Motion Reconstruction [68.1355376294881]
  We present a large-scale dataset synchronizing consumer-grade sensors with ground-truth 3D motion.<n>Second, we propose WHIP, a baseline generative model that reconstructs motion from arbitrary subsets of available sensors.<n>Third, we conduct a systematic study of sensor complementarity, quantifying how different modalities complement one another.
  arXiv  Detail & Related papers  (2026-07-08T11:56:02Z)
• Integrated Sensing and Communications for Real-time Avatar Control in XR over 5G [3.8468982193073757]
  Extended Reality (XR) presents a challenging use case for 5G and 6G networks.<n>Current XR interaction solutions based on handheld controllers and cameras cannot easily capture full-body poses.<n>We propose a multimodal sensing architecture for XR that combines 5G MillimeterWave (mmWave) Integrated sensing and communication (ISAC) and surface electromyography (sEMG) signals.
  arXiv  Detail & Related papers  (2026-06-22T17:27:42Z)
• EgoEMG: A Multimodal Egocentric Dataset with Bilateral EMG and Vision for Hand Pose Estimation [21.77386351667022]
  EgoEMG is a multimodal egocentric dataset for bimanual hand pose estimation.<n>The dataset covers 41 participants performing 60 gesture classes, including 30 single-hand gestures and 30 bimanual gestures.<n>EgoEMG and its benchmark establish a foundation for future research on multimodal hand pose estimation with EMG and vision.
  arXiv  Detail & Related papers  (2026-05-07T05:56:57Z)
• DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization [5.972698171130012]
  High-fidelity teleoperation of dexterous robotic hands is essential for bringing robots into unstructured domestic environments.<n>We develop a lightweight and cost-effective teleoperation system leveraging surface electromyography (sEMG)<n>We show that our system achieves high precision in diverse teleoperation tasks.
  arXiv  Detail & Related papers  (2026-03-06T03:36:33Z)
• MobileManiBench: Simplifying Model Verification for Mobile Manipulation [70.30578259859512]
  MobileManiBench is a large-scale benchmark for mobile-based robotic manipulation.<n>MobileManiBench features 2 mobile platforms (parallel-gripper and dexterous-hand robots), 2 synchronized cameras (head and right wrist), 630 objects in 20 categories, 5 skills (open, close, pull, push, pick) with over 100 tasks performed in 100 realistic scenes, yielding 300K trajectories.
  arXiv  Detail & Related papers  (2026-02-05T02:49:52Z)
• A Comparative Study of EMG- and IMU-based Gesture Recognition at the Wrist and Forearm [3.990794855710089]
  IMU signals contain sufficient information to serve as the sole input sensor for static gesture recognition.<n> tendon-induced micro-movement captured by IMUs is a major contributor to static gesture recognition.
  arXiv  Detail & Related papers  (2025-12-08T19:36:10Z)
• Recognizing Actions from Robotic View for Natural Human-Robot Interaction [52.00935005918032]
  Natural Human-Robot Interaction (N-HRI) requires robots to recognize human actions at varying distances and states, regardless of whether the robot itself is in motion or stationary.<n>Existing benchmarks for N-HRI fail to address the unique complexities in N-HRI due to limited data, modalities, task categories, and diversity of subjects and environments.<n>We introduce (Action from Robotic View) a large-scale dataset for perception-centric robotic views prevalent in mobile service robots.
  arXiv  Detail & Related papers  (2025-07-30T09:48:34Z)
• emg2tendon: From sEMG Signals to Tendon Control in Musculoskeletal Hands [5.613626927694011]
  Tendon-driven robotic hands offer unparalleled dexterity for manipulation tasks.<n>However, learning control policies for such systems presents unique challenges.<n>We introduce the first large-scale EMG-to-Tendon Control dataset for robotic hands.
  arXiv  Detail & Related papers  (2025-07-29T12:49:57Z)
• EgoHand: Ego-centric Hand Pose Estimation and Gesture Recognition with Head-mounted Millimeter-wave Radar and IMUs [15.644891766887255]
  Bottom-facing VR cameras can pose a risk of exposing sensitive information, such as private body parts or personal surroundings. We introduce EgoHand, a system that integrates millimeter-wave radar and IMUs for hand gesture recognition. In experiments, EgoHand can detect hand gestures with 90.8% accuracy.
  arXiv  Detail & Related papers  (2025-01-23T16:25:08Z)
• emg2qwerty: A Large Dataset with Baselines for Touch Typing using Surface Electromyography [47.160223334501126]
  emg2qwerty is a large-scale dataset of non-invasive electromyographic signals recorded at the wrists while touch typing on a QWERTY keyboard. With 1,135 sessions spanning 108 users and 346 hours of recording, this is the largest such public dataset to date. We show strong baseline performance on predicting key-presses using sEMG signals alone.
  arXiv  Detail & Related papers  (2024-10-26T05:18:48Z)
• EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision [69.1005706608681]
  We introduce EgoPressure, a novel dataset of touch contact and pressure interaction from an egocentric perspective. EgoPressure comprises 5.0 hours of touch contact and pressure interaction from 21 participants captured by a moving egocentric camera and 7 stationary Kinect cameras.
  arXiv  Detail & Related papers  (2024-09-03T18:53:32Z)
• FORS-EMG: A Novel sEMG Dataset for Hand Gesture Recognition Across Multiple Forearm Orientations [1.3852370777848657]
  Surface electromyography (sEMG) signals hold significant potential for gesture recognition and robust prosthetic hand development. This study introduces a novel MFI sEMG dataset to evaluate commonly used hand gestures across three distinct orientations.
  arXiv  Detail & Related papers  (2024-09-03T14:23:06Z)
• Motion Capture from Inertial and Vision Sensors [73.3003181330859]
  We contribute MINIONS, a large-scale Motion capture dataset collected from INertial and visION Sensors.<n>With the proposed MINIONS dataset, we propose a SparseNet framework to capture human motion from IMUs and videos.<n>Experiment results emphasize the unique advantages of inertial and vision sensors, showcasing the promise of consumer-affordable multi-modal motion capture.
  arXiv  Detail & Related papers  (2024-07-23T09:41:10Z)
• Decomposed Vector-Quantized Variational Autoencoder for Human Grasp Generation [27.206656215734295]
  We propose a novel Decomposed Vector-Quantized Variational Autoencoder (DVQ-VAE) to generate realistic human grasps. Part-aware decomposed architecture facilitates more precise management of the interaction between each component of hand and object. Our model achieved about 14.1% relative improvement in the quality index compared to the state-of-the-art methods in four widely-adopted benchmarks.
  arXiv  Detail & Related papers  (2024-07-19T06:41:16Z)
• Learning Visuotactile Skills with Two Multifingered Hands [80.99370364907278]
  We explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data.
  arXiv  Detail & Related papers  (2024-04-25T17:59:41Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• EMG subspace alignment and visualization for cross-subject hand gesture classification [0.125828876338076]
  The paper discusses and analyses the challenge of cross-subject generalization thanks to an original dataset containing the EMG signals of 14 human subjects during hand gestures. The experimental results show that, though an accurate generalization based on pooling multiple subjects is hardly achievable, it is possible to improve the cross-subject estimation by identifying a robust low-dimensional subspace for multiple subjects and aligning it to a target subject.
  arXiv  Detail & Related papers  (2023-12-18T14:32:29Z)
• Agile gesture recognition for capacitive sensing devices: adapting on-the-job [55.40855017016652]
  We demonstrate a hand gesture recognition system that uses signals from capacitive sensors embedded into the etee hand controller. The controller generates real-time signals from each of the wearer five fingers. We use a machine learning technique to analyse the time series signals and identify three features that can represent 5 fingers within 500 ms.
  arXiv  Detail & Related papers  (2023-05-12T17:24:02Z)
• Reconfigurable Data Glove for Reconstructing Physical and Virtual Grasps [100.72245315180433]
  We present a reconfigurable data glove design to capture different modes of human hand-object interactions. The glove operates in three modes for various downstream tasks with distinct features. We evaluate the system's three modes by (i) recording hand gestures and associated forces, (ii) improving manipulation fluency in VR, and (iii) producing realistic simulation effects of various tool uses.
  arXiv  Detail & Related papers  (2023-01-14T05:35:50Z)
• ViT-HGR: Vision Transformer-based Hand Gesture Recognition from High Density Surface EMG Signals [14.419091034872682]
  We investigate and design a Vision Transformer (ViT) based architecture to perform hand gesture recognition from High Density (HD-sEMG) signals. The proposed ViT-HGR framework can overcome the training time problems and can accurately classify a large number of hand gestures from scratch. Our experiments with 64-sample (31.25 ms) window size yield average test accuracy of 84.62 +/- 3.07%, where only 78, 210 number of parameters is utilized.
  arXiv  Detail & Related papers  (2022-01-25T02:42:50Z)
• HANDS: A Multimodal Dataset for Modeling Towards Human Grasp Intent Inference in Prosthetic Hands [3.7886097009023376]
  Advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user. multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control.
  arXiv  Detail & Related papers  (2021-03-08T15:51:03Z)
• Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning [52.37106940303246]
  We learn a model that maps noisy input hand poses to target virtual poses. The agent is trained in a residual setting by using a model-free hybrid RL+IL approach. We test our framework in two applications that use hand pose estimates for dexterous manipulations: hand-object interactions in VR and hand-object motion reconstruction in-the-wild.
  arXiv  Detail & Related papers  (2020-08-07T17:34:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.