STAR: Smartphone-analogous Typing in Augmented Reality

• URL: http://arxiv.org/abs/2511.21143v1
• Date: Wed, 26 Nov 2025 07:53:09 GMT
• Title: STAR: Smartphone-analogous Typing in Augmented Reality
• Authors: Taejun Kim, Amy Karlson, Aakar Gupta, Tovi Grossman, Jason Wu, Parastoo Abtahi, Christopher Collins, Michael Glueck, Hemant Bhaskar Surale,
• Abstract summary: This research presents STAR, a smartphone-analogous AR text entry technique.<n>With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands.<n>During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM, and a mean error rate of 0.3% after 30 minutes of practice.
• Score: 22.217722884498087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While text entry is an essential and frequent task in Augmented Reality (AR) applications, devising an efficient and easy-to-use text entry method for AR remains an open challenge. This research presents STAR, a smartphone-analogous AR text entry technique that leverages a user's familiarity with smartphone two-thumb typing. With STAR, a user performs thumb typing on a virtual QWERTY keyboard that is overlain on the skin of their hands. During an evaluation study of STAR, participants achieved a mean typing speed of 21.9 WPM (i.e., 56% of their smartphone typing speed), and a mean error rate of 0.3% after 30 minutes of practice. We further analyze the major factors implicated in the performance gap between STAR and smartphone typing, and discuss ways this gap could be narrowed.

Related papers

• RingGesture: A Ring-Based Mid-Air Gesture Typing System Powered by a Deep-Learning Word Prediction Framework [2.4992122541451987]
  RingGesture is a ring-based mid-air gesture typing technique utilizing electrodes to mark the start and end of gesture trajectories. We propose a novel deep-learning word prediction framework, Score Fusion, comprised of three key components. RingGesture achieves an average text entry speed of 27.3 words per minute (WPM) and a peak performance of 47.9 WPM.
  arXiv  Detail & Related papers  (2024-10-08T13:15:30Z)
• TapType: Ten-finger text entry on everyday surfaces via Bayesian inference [32.33746932895968]
  TapType is a mobile text entry system for full-size typing on passive surfaces. From the inertial sensors inside a band on either wrist, TapType decodes and relates surface taps to a traditional QWERTY keyboard layout.
  arXiv  Detail & Related papers  (2024-10-08T12:58:31Z)
• TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision [25.271209425555906]
  We present a real-time pipeline that detects touch input from all ten fingers on any physical surface. Our method TouchInsight comprises a neural network to predict the moment of a touch event, the finger making contact, and the touch location.
  arXiv  Detail & Related papers  (2024-10-08T11:42:44Z)
• Using Large Language Models to Accelerate Communication for Users with Severe Motor Impairments [17.715162857028595]
  We present SpeakFaster, consisting of large language models (LLMs) and a co-designed user interface for text entry in a highly-abbreviated form. Pilot study with 19 non-AAC participants typing on a mobile device by hand demonstrated gains in motor savings in line with the offline simulation. Lab and field testing on two eye-gaze typing users with amyotrophic lateral sclerosis (ALS) demonstrated text-entry rates 29-60% faster than traditional baselines.
  arXiv  Detail & Related papers  (2023-12-03T23:12:49Z)
• Generative Input: Towards Next-Generation Input Methods Paradigm [49.98958865125018]
  We propose a novel Generative Input paradigm named GeneInput. It uses prompts to handle all input scenarios and other intelligent auxiliary input functions, optimizing the model with user feedback to deliver personalized results. The results demonstrate that we have achieved state-of-the-art performance for the first time in the Full-mode Key-sequence to Characters(FK2C) task.
  arXiv  Detail & Related papers  (2023-11-02T12:01:29Z)
• Typing on Any Surface: A Deep Learning-based Method for Real-Time Keystroke Detection in Augmented Reality [4.857109990499532]
  Mid-air keyboard interface, wireless keyboards or voice input, either suffer from poor ergonomic design, limited accuracy, or are simply embarrassing to use in public. This paper proposes and validates a deep-learning based approach, that enables AR applications to accurately predict keystrokes from the user perspective RGB video stream. A two-stage model, combing an off-the-shelf hand landmark extractor and a novel adaptive Convolutional Recurrent Neural Network (C-RNN) was trained.
  arXiv  Detail & Related papers  (2023-08-31T23:58:25Z)
• Multi-modal Queried Object Detection in the Wild [72.16067634379226]
  MQ-Det is an efficient architecture and pre-training strategy design for real-world object detection. It incorporates vision queries into existing language-queried-only detectors. MQ-Det's simple yet effective architecture and training strategy design is compatible with most language-queried object detectors.
  arXiv  Detail & Related papers  (2023-05-30T12:24:38Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• Effidit: Your AI Writing Assistant [60.588370965898534]
  Effidit is a digital writing assistant that facilitates users to write higher-quality text more efficiently by using artificial intelligence (AI) technologies. In Effidit, we significantly expand the capacities of a writing assistant by providing functions in five categories: text completion, error checking, text polishing, keywords to sentences (K2S), and cloud input methods (cloud IME)
  arXiv  Detail & Related papers  (2022-08-03T02:24:45Z)
• Mobile Behavioral Biometrics for Passive Authentication [65.94403066225384]
  This work carries out a comparative analysis of unimodal and multimodal behavioral biometric traits. Experiments are performed over HuMIdb, one of the largest and most comprehensive freely available mobile user interaction databases. In our experiments, the most discriminative background sensor is the magnetometer, whereas among touch tasks the best results are achieved with keystroke.
  arXiv  Detail & Related papers  (2022-03-14T17:05:59Z)
• DeepMark++: Real-time Clothing Detection at the Edge [55.41644538483948]
  We propose a single-stage approach to deliver rapid clothing detection and keypoint estimation. Our solution is based on a multi-target network CenterNet, and we introduce several powerful post-processing techniques to enhance performance. Our most accurate model achieves results comparable to state-of-the-art solutions on the DeepFashion2 dataset.
  arXiv  Detail & Related papers  (2020-06-01T04:36:57Z)
• Towards Palmprint Verification On Smartphones [62.279124220123286]
  Studies in the past two decades have shown that palmprints have outstanding merits in uniqueness and permanence. We built a DCNN-based palmprint verification system named DeepMPV+ for smartphones. The efficiency and efficacy of DeepMPV+ have been corroborated by extensive experiments.
  arXiv  Detail & Related papers  (2020-03-30T08:31:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.