Accessible Gesture-Driven Augmented Reality Interaction System

• URL: http://arxiv.org/abs/2506.15189v1
• Date: Wed, 18 Jun 2025 07:10:48 GMT
• Title: Accessible Gesture-Driven Augmented Reality Interaction System
• Authors: Yikan Wang,
• Abstract summary: Augmented reality (AR) offers immersive interaction but remains inaccessible for users with motor impairments or limited dexterity.<n>This study proposes a gesture-based interaction system for AR environments, leveraging deep learning to recognize hand and body gestures.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Augmented reality (AR) offers immersive interaction but remains inaccessible for users with motor impairments or limited dexterity due to reliance on precise input methods. This study proposes a gesture-based interaction system for AR environments, leveraging deep learning to recognize hand and body gestures from wearable sensors and cameras, adapting interfaces to user capabilities. The system employs vision transformers (ViTs), temporal convolutional networks (TCNs), and graph attention networks (GATs) for gesture processing, with federated learning ensuring privacy-preserving model training across diverse users. Reinforcement learning optimizes interface elements like menu layouts and interaction modes. Experiments demonstrate a 20% improvement in task completion efficiency and a 25% increase in user satisfaction for motor-impaired users compared to baseline AR systems. This approach enhances AR accessibility and scalability. Keywords: Deep learning, Federated learning, Gesture recognition, Augmented reality, Accessibility, Human-computer interaction

Related papers

• Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection [51.52749744031413]
  Human-Object Interaction (HOI) detection aims to identify humans and objects within images and interpret their interactions.<n>Existing HOI methods rely heavily on large datasets with manual annotations to learn interactions from visual cues.<n>We propose a novel training-free HOI detection framework for Dynamic Scoring with enhanced semantics.
  arXiv  Detail & Related papers  (2025-07-23T12:30:19Z)
• Fatigue-Aware Adaptive Interfaces for Wearable Devices Using Deep Learning [0.0]
  This study proposes a fatigue-aware adaptive interface system for wearable devices.<n>It uses deep learning to analyze physiological data and adjust interface elements to mitigate cognitive load.<n> Experimental results show a 18% reduction in cognitive load and a 22% improvement in user satisfaction.
  arXiv  Detail & Related papers  (2025-06-16T08:07:07Z)
• DiG-Net: Enhancing Quality of Life through Hyper-Range Dynamic Gesture Recognition in Assistive Robotics [2.625826951636656]
  We introduce a novel approach designed specifically for assistive robotics, enabling dynamic gesture recognition at extended distances of up to 30 meters.<n>Our proposed Distance-aware Gesture Network (DiG-Net) effectively combines Depth-Conditioned Deformable Alignment (DADA) blocks with Spatio-Temporal Graph modules.<n>By effectively interpreting gestures from considerable distances, DiG-Net significantly enhances the usability of assistive robots in home healthcare, industrial safety, and remote assistance scenarios.
  arXiv  Detail & Related papers  (2025-05-30T16:47:44Z)
• Computer Vision-Driven Gesture Recognition: Toward Natural and Intuitive Human-Computer [21.70275919660522]
  This study explores the application of natural gesture recognition based on computer vision in human-computer interaction.<n>By connecting the palm and each finger joint, a dynamic and static gesture model of the hand is formed.<n> Experimental results show that this method can effectively recognize various gestures and maintain high recognition accuracy and real-time response capabilities.
  arXiv  Detail & Related papers  (2024-12-24T10:13:20Z)
• Learning Manipulation by Predicting Interaction [85.57297574510507]
  We propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction. The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms.
  arXiv  Detail & Related papers  (2024-06-01T13:28:31Z)
• Bootstrapping Adaptive Human-Machine Interfaces with Offline Reinforcement Learning [82.91837418721182]
  Adaptive interfaces can help users perform sequential decision-making tasks. Recent advances in human-in-the-loop machine learning enable such systems to improve by interacting with users. We propose a reinforcement learning algorithm to train an interface to map raw command signals to actions.
  arXiv  Detail & Related papers  (2023-09-07T16:52:27Z)
• Force-Aware Interface via Electromyography for Natural VR/AR Interaction [69.1332992637271]
  We design a learning-based neural interface for natural and intuitive force inputs in VR/AR. We show that our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. We envision our findings to push forward research towards more realistic physicality in future VR/AR.
  arXiv  Detail & Related papers  (2022-10-03T20:51:25Z)
• GesSure -- A Robust Face-Authentication enabled Dynamic Gesture Recognition GUI Application [1.3649494534428745]
  This paper aims to design a robust, face-verification-enabled gesture recognition system. We use meaningful and relevant gestures for task operation, resulting in a better user experience. Our prototype has successfully executed context-dependent tasks like save, print, control video-player operations and exit, and context-free operating system tasks like sleep, shut-down, and unlock intuitively.
  arXiv  Detail & Related papers  (2022-07-22T12:14:35Z)
• ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning [91.58711082348293]
  Reinforcement learning from online user feedback on the system's performance presents a natural solution to this problem. This approach tends to require a large amount of human-in-the-loop training data, especially when feedback is sparse. We propose a hierarchical solution that learns efficiently from sparse user feedback.
  arXiv  Detail & Related papers  (2022-02-05T02:01:19Z)
• Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition [54.749127627191655]
  The ability to recognize human partners is an important social skill to build personalized and long-term human-robot interactions. Deep learning networks have achieved state-of-the-art results and demonstrated to be suitable tools to address such a task. One solution is to make robots learn from their first-hand sensory data with self-supervision.
  arXiv  Detail & Related papers  (2021-03-16T13:50:24Z)
• Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognition [131.6328804788164]
  We propose a framework, named Semantics-aware Adaptive Knowledge Distillation Networks (SAKDN), to enhance action recognition in vision-sensor modality (videos) The SAKDN uses multiple wearable-sensors as teacher modalities and uses RGB videos as student modality.
  arXiv  Detail & Related papers  (2020-09-01T03:38:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.