THETA: Triangulated Hand-State Estimation for Teleoperation and Automation in Robotic Hand Control

• URL: http://arxiv.org/abs/2601.07768v1
• Date: Mon, 12 Jan 2026 17:50:02 GMT
• Title: THETA: Triangulated Hand-State Estimation for Teleoperation and Automation in Robotic Hand Control
• Abstract summary: We present a novel, cost-effective approach using three webcams for triangulation-based tracking to approximate relative joint angles (theta) of human fingers.<n>In real-time inference, THETA captures simultaneous frames, segments hand regions, filters them, and compiles a 9-channel tensor for classification.<n> THETA potentially ensures cost-effective, user-friendly teleoperation for medical, linguistic, and manufacturing applications.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The teleoperation of robotic hands is limited by the high costs of depth cameras and sensor gloves, commonly used to estimate hand relative joint positions (XYZ). We present a novel, cost-effective approach using three webcams for triangulation-based tracking to approximate relative joint angles (theta) of human fingers. We also introduce a modified DexHand, a low-cost robotic hand from TheRobotStudio, to demonstrate THETA's real-time application. Data collection involved 40 distinct hand gestures using three 640x480p webcams arranged at 120-degree intervals, generating over 48,000 RGB images. Joint angles were manually determined by measuring midpoints of the MCP, PIP, and DIP finger joints. Captured RGB frames were processed using a DeepLabV3 segmentation model with a ResNet-50 backbone for multi-scale hand segmentation. The segmented images were then HSV-filtered and fed into THETA's architecture, consisting of a MobileNetV2-based CNN classifier optimized for hierarchical spatial feature extraction and a 9-channel input tensor encoding multi-perspective hand representations. The classification model maps segmented hand views into discrete joint angles, achieving 97.18% accuracy, 98.72% recall, F1 Score of 0.9274, and a precision of 0.8906. In real-time inference, THETA captures simultaneous frames, segments hand regions, filters them, and compiles a 9-channel tensor for classification. Joint-angle predictions are relayed via serial to an Arduino, enabling the DexHand to replicate hand movements. Future research will increase dataset diversity, integrate wrist tracking, and apply computer vision techniques such as OpenAI-Vision. THETA potentially ensures cost-effective, user-friendly teleoperation for medical, linguistic, and manufacturing applications.

Related papers

• DreamHand: Repurposing Video Diffusion Models for Occlusion-Robust Egocentric 3D Hand Motion Recovery [51.92897182932217]
  Egocentric video offers scalable manipulation data for embodied AI.<n>However, recovering metric 3D hand trajectories is challenging due to severe object occlusion and frequent out-of-sight gaps.<n>We introduce DreamHand, an offline clip-level framework that extracts features via a Deterministic Clean-Latent and decodes them with a Bidirectional Spatiotemporal Decoder.
  arXiv  Detail & Related papers  (2026-08-20T17:46:24Z)
• Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics [0.0]
  We present an offline hand-shadowing and mapping pipeline from a single egocentric RGB-D camera mounted on 3D-printed glasses.<n>The pipeline detects 21 hand landmarks per hand using MediaPipe Hands.<n>A gripper controller maps thumb-index finger geometry to grasp aperture with a four-level fallback hierarchy.
  arXiv  Detail & Related papers  (2026-03-11T23:53:28Z)
• Cross-Embodiment Dexterous Hand Articulation Generation via Morphology-Aware Learning [82.63833405368159]
  Existing end-to-end methods require training on large-scale datasets for specific hands.<n>We propose an eigengrasp-based, end-to-end framework for cross-embodiment grasp generation.
  arXiv  Detail & Related papers  (2025-10-07T15:57:00Z)
• WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild [53.288327629960364]
  We present a data-driven pipeline for efficient multi-hand reconstruction in the wild.<n>The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model.<n>Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks.
  arXiv  Detail & Related papers  (2024-09-18T18:46:51Z)
• Simultaneous prediction of hand gestures, handedness, and hand keypoints using thermal images [0.6087960723103347]
  We propose a technique for simultaneous hand gesture classification, handedness detection, and hand keypoints localization using thermal data captured by an infrared camera. Our method uses a novel deep multi-task learning architecture that includes shared encoderdecoder layers followed by three branches dedicated for each mentioned task.
  arXiv  Detail & Related papers  (2023-03-02T19:25:40Z)
• Simultaneous Estimation of Hand Configurations and Finger Joint Angles using Forearm Ultrasound [8.753262480814493]
  Forearm ultrasound images provide a musculoskeletal visualization that can be used to understand hand motion. We propose a CNN based deep learning pipeline for predicting the MCP joint angles. A low latency pipeline has been proposed for estimating both MCP joint angles and hand configuration aimed at real-time control of human-machine interfaces.
  arXiv  Detail & Related papers  (2022-11-29T02:06:19Z)
• Hand gesture recognition using 802.11ad mmWave sensor in the mobile device [2.5476515662939563]
  We explore the feasibility of AI assisted hand-gesture recognition using 802.11ad 60GHz (mmWave) technology in smartphones. We built a prototype system, where radar sensing and communication waveform can coexist by time-division duplex (TDD) It can gather sensing data and predict gestures within 100 milliseconds.
  arXiv  Detail & Related papers  (2022-11-14T03:36:17Z)
• HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks [71.09275975580009]
  HandVoxNet++ is a voxel-based deep network with 3D and graph convolutions trained in a fully supervised manner. HandVoxNet++ relies on two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which
  arXiv  Detail & Related papers  (2021-07-02T17:59:54Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image [71.17227941339935]
  We propose a large-scale dataset, InterHand2.6M, and a network, InterNet, for 3D interacting hand pose estimation from a single RGB image. In our experiments, we demonstrate big gains in 3D interacting hand pose estimation accuracy when leveraging the interacting hand data in InterHand2.6M. We also report the accuracy of InterNet on InterHand2.6M, which serves as a strong baseline for this new dataset.
  arXiv  Detail & Related papers  (2020-08-21T05:15:58Z)
• IPN Hand: A Video Dataset and Benchmark for Real-Time Continuous Hand Gesture Recognition [11.917058689674327]
  We introduce a new benchmark dataset named IPN Hand with sufficient size, variety, and real-world elements able to train and evaluate deep neural networks. This dataset contains more than 4,000 gesture samples and 800,000 RGB frames from 50 distinct subjects. With our dataset, the performance of three 3D-CNN models is evaluated on the tasks of isolated and continuous real-time HGR.
  arXiv  Detail & Related papers  (2020-04-20T08:52:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.