V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction

• URL: http://arxiv.org/abs/2409.13347v1
• Date: Fri, 20 Sep 2024 09:26:53 GMT
• Title: V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction
• Abstract summary: In whiteboard-based remote communication, the seamless integration of drawn content and hand-screen interactions is essential. Previous methods either require bulky device setups for capturing hand gestures or fail to accurately track the hand poses from capacitive images. We present a real-time method for precise tracking 3D poses of both hands from capacitive video frames.
• Score: 11.808862837391567
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In whiteboard-based remote communication, the seamless integration of drawn content and hand-screen interactions is essential for an immersive user experience. Previous methods either require bulky device setups for capturing hand gestures or fail to accurately track the hand poses from capacitive images. In this paper, we present a real-time method for precise tracking 3D poses of both hands from capacitive video frames. To this end, we develop a deep neural network to identify hands and infer hand joint positions from capacitive frames, and then recover 3D hand poses from the hand-joint positions via a constrained inverse kinematic solver. Additionally, we design a device setup for capturing high-quality hand-screen interaction data and obtained a more accurate synchronized capacitive video and hand pose dataset. Our method improves the accuracy and stability of 3D hand tracking for capacitive frames while maintaining a compact device setup for remote communication. We validate our scheme design and its superior performance on 3D hand pose tracking and demonstrate the effectiveness of our method in whiteboard-based remote communication. Our code, model, and dataset are available at https://V-Hands.github.io.

Related papers

• HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control [51.913695868638634]
  We present HandsOnWorld, a framework for hand-controlled egocentric video generation that forgoes multi-view and marker-based motion capture, learning instead from unconstrained monocular video.<n>We build EgoVid-Pro, a dataset of clean, protagonist-only hand trajectories spanning 103K clips and roughly 12M frames across diverse everyday scenes.<n>We also propose the Plcker Hand Map, a 3D-aware control signal that extends Plcker-ray representations from camera rays to the hand surface, disentangling camera and hand motion at the representation level.
  arXiv  Detail & Related papers  (2026-07-02T12:14:02Z)
• Gesture-Aware Pretraining and Token Fusion for 3D Hand Pose Estimation [3.0979829079231447]
  We show that gesture semantics can serve as a powerful inductive bias for 3D pose estimation.<n>We present a two-stage framework: gesture-aware pretraining that learns an informative embedding space.<n>Experiments on InterHand2.6M demonstrate that gesture-aware pretraining consistently improves single-hand accuracy.
  arXiv  Detail & Related papers  (2026-03-18T06:18:46Z)
• AssemblyHands-X: Modeling 3D Hand-Body Coordination for Understanding Bimanual Human Activities [27.634829042887358]
  We present AssemblyHands-X, the first markerless 3D hand-body benchmark for bimanual activities.<n>Our approach combines multi-view triangulation with SMPL-X mesh fitting, yielding reliable 3D registration of hands and upper body.<n>Our experiments show pose-based action inference is more efficient and accurate than video baselines.
  arXiv  Detail & Related papers  (2025-09-28T13:52:14Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• RemoteTouch: Enhancing Immersive 3D Video Communication with Hand Touch [28.096602057231035]
  In our system, each meeting participant sits in front of a large screen with haptic feedback. The local participant can reach his hand out to the screen and perform hand clapping with the remote participant as if the two participants were only separated by a virtual glass.
  arXiv  Detail & Related papers  (2023-02-28T07:37:53Z)
• LG-Hand: Advancing 3D Hand Pose Estimation with Locally and Globally Kinematic Knowledge [0.693939291118954]
  We propose LG-Hand, a powerful method for 3D hand pose estimation. We argue that kinematic information plays an important role, contributing to the performance of 3D hand pose estimation. Our method achieves promising results on the First-Person Hand Action Benchmark dataset.
  arXiv  Detail & Related papers  (2022-11-06T15:26:32Z)
• Consistent 3D Hand Reconstruction in Video via self-supervised Learning [67.55449194046996]
  We present a method for reconstructing accurate and consistent 3D hands from a monocular video. detected 2D hand keypoints and the image texture provide important cues about the geometry and texture of the 3D hand. We propose $rm S2HAND$, a self-supervised 3D hand reconstruction model.
  arXiv  Detail & Related papers  (2022-01-24T09:44:11Z)
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements [96.40125818594952]
  We make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions.
  arXiv  Detail & Related papers  (2021-11-01T08:24:10Z)
• Egocentric View Hand Action Recognition by Leveraging Hand Surface and Hand Grasp Type [15.878905144552204]
  The framework synthesizes the mean curvature of the hand mesh model to encode the hand surface geometry in 3D space. Using hand grasp type and mean curvature of hand increases the performance of the hand action recognition.
  arXiv  Detail & Related papers  (2021-09-08T17:12:02Z)
• H2O: Two Hands Manipulating Objects for First Person Interaction Recognition [70.46638409156772]
  We present a comprehensive framework for egocentric interaction recognition using markerless 3D annotations of two hands manipulating objects. Our method produces annotations of the 3D pose of two hands and the 6D pose of the manipulated objects, along with their interaction labels for each frame. Our dataset, called H2O (2 Hands and Objects), provides synchronized multi-view RGB-D images, interaction labels, object classes, ground-truth 3D poses for left & right hands, 6D object poses, ground-truth camera poses, object meshes and scene point clouds.
  arXiv  Detail & Related papers  (2021-04-22T17:10:42Z)
• MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis [81.40640219844197]
  Estimating the 3D hand pose from a monocular RGB image is important but challenging. A solution is training on large-scale RGB hand images with accurate 3D hand keypoint annotations. We have developed a learning-based approach to synthesize realistic, diverse, and 3D pose-preserving hand images.
  arXiv  Detail & Related papers  (2020-10-02T18:27:34Z)
• Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics [87.17505994436308]
  We build upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Our hand prediction model produces convincing 3D hand gestures given only the 3D motion of the speaker's arms as input.
  arXiv  Detail & Related papers  (2020-07-23T22:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.