Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface

• URL: http://arxiv.org/abs/2601.15209v2
• Date: Thu, 22 Jan 2026 16:01:25 GMT
• Title: Deaf and Hard of Hearing Access to Intelligent Personal Assistants: Comparison of Voice-Based Options with an LLM-Powered Touch Interface
• Abstract summary: We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people.<n>The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals.
• Score: 7.968583771762071
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate intelligent personal assistants (IPAs) accessibility for deaf and hard of hearing (DHH) people who can use their voice in everyday communication. The inability of IPAs to understand diverse accents including deaf speech renders them largely inaccessible to non-signing and speaking DHH individuals. Using an Echo Show, we compare the usability of natural language input via spoken English; with Alexa's automatic speech recognition and a Wizard-of-Oz setting with a trained facilitator re-speaking commands against that of a large language model (LLM)-assisted touch interface in a mixed-methods study. The touch method was navigated through an LLM-powered "task prompter," which integrated the user's history and smart environment to suggest contextually-appropriate commands. Quantitative results showed no significant differences across both spoken English conditions vs LLM-assisted touch. Qualitative results showed variability in opinions on the usability of each method. Ultimately, it will be necessary to have robust deaf-accented speech recognized natively by IPAs.

Related papers

• From Silent Signals to Natural Language: A Dual-Stage Transformer-LLM Approach [0.0]
  We propose an automatic speech recognition framework that combines a transformer-based acoustic model with a large language model (LLM) for post-processing.<n> Experimental results show a 16% relative and 6% absolute reduction in word error rate (WER) over a 36% baseline, demonstrating substantial improvements in intelligibility for silent speech interfaces.
  arXiv  Detail & Related papers  (2025-09-02T16:13:29Z)
• Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio [52.859261069569165]
  We propose the first unified framework capable of handling diverse combinations of sign language, lip movements, and audio for spoken-language text generation.<n>We focus on three main objectives: (i) designing a unified, modality-agnostic architecture capable of effectively processing heterogeneous inputs; (ii) exploring the underexamined synergy among modalities, particularly the role of lip movements as non-manual cues in sign language comprehension; and (iii) achieving performance on par with or better than state-of-the-art models specialized for individual tasks.
  arXiv  Detail & Related papers  (2025-08-28T06:51:42Z)
• Real-Time Sign Language Gestures to Speech Transcription using Deep Learning [0.0]
  This project introduces a real-time assistive technology solution that leverages advanced deep learning techniques to translate sign language gestures into textual and audible speech.<n>By employing convolution neural networks (CNN) trained on the Sign Language MNIST dataset, the system accurately classifies hand gestures captured live via webcam.
  arXiv  Detail & Related papers  (2025-08-18T08:25:18Z)
• SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models [76.07833875692722]
  Speech-based Intelligence Quotient (SIQ) is a new form of human cognition-inspired evaluation pipeline for voice understanding large language models.<n>Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks.
  arXiv  Detail & Related papers  (2025-07-25T15:12:06Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Audio-Aware Large Language Models as Judges for Speaking Styles [123.36224336701237]
  We explore using audio-aware large language models (ALLMs) as an automatic judge to assess the speaking styles of speeches.<n>We use four spoken language models (SLMs) to complete the two tasks and use humans and ALLMs to judge the SLMs' responses.<n>Our results show that current SLMs, even GPT-4o-audio, still have room for improvement in controlling the speaking style and generating natural dialogues.
  arXiv  Detail & Related papers  (2025-06-06T11:05:48Z)
• Your voice is your voice: Supporting Self-expression through Speech Generation and LLMs in Augmented and Alternative Communication [9.812902134556971]
  Speak Ease is an augmentative and alternative communication system to support users' expressivity.<n>System integrates multimodal input, including text, voice, and contextual cues, with large language models.
  arXiv  Detail & Related papers  (2025-03-21T18:50:05Z)
• VoiceBench: Benchmarking LLM-Based Voice Assistants [58.84144494938931]
  We introduce VoiceBench, the first benchmark to evaluate voice assistants based on large language models (LLMs)<n>VoiceBench includes both real and synthetic spoken instructions that incorporate the above three key real-world variations.<n>Extensive experiments reveal the limitations of current LLM-based voice assistant models and offer valuable insights for future research and development in this field.
  arXiv  Detail & Related papers  (2024-10-22T17:15:20Z)
• Distilling an End-to-End Voice Assistant Without Instruction Training Data [53.524071162124464]
  Distilled Voice Assistant (DiVA) generalizes to Question Answering, Classification, and Translation. We show that DiVA better meets user preferences, achieving a 72% win rate compared with state-of-the-art models like Qwen 2 Audio.
  arXiv  Detail & Related papers  (2024-10-03T17:04:48Z)
• Evaluation of Automated Speech Recognition Systems for Conversational Speech: A Linguistic Perspective [0.0]
  We take a linguistic perspective, and take the French language as a case study toward disambiguation of the French homophones. Our contribution aims to provide more insight into human speech transcription accuracy in conditions to reproduce those of state-of-the-art ASR systems.
  arXiv  Detail & Related papers  (2022-11-05T04:35:40Z)
• See what I'm saying? Comparing Intelligent Personal Assistant use for Native and Non-Native Language Speakers [12.021912017203842]
  We aim to understand how IPAs currently support or hinder users. Interviews revealed that L2 speakers prioritised utterance planning around perceived linguistic limitations. L2 speakers see IPAs as insensitive to linguistic needs resulting in failed interaction. We discuss the need to tailor the IPA experience for L2 users, emphasising visual feedback whilst reducing the burden of language production.
  arXiv  Detail & Related papers  (2020-06-11T11:03:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.