Sanvaad: A Multimodal Accessibility Framework for ISL Recognition and Voice-Based Interaction

• URL: http://arxiv.org/abs/2512.06485v1
• Date: Sat, 06 Dec 2025 16:16:21 GMT
• Title: Sanvaad: A Multimodal Accessibility Framework for ISL Recognition and Voice-Based Interaction
• Abstract summary: Sanvaad is a lightweight multimodal accessibility framework designed to support real time, two-way communication.<n>For deaf users, Sanvaad includes an ISL recognition module built on MediaPipe landmarks.<n>For visually impaired users, the framework provides a screen free voice interface that integrates multilingual speech recognition, text summarization, and text-to-speech generation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Communication between deaf users, visually im paired users, and the general hearing population often relies on tools that support only one direction of interaction. To address this limitation, this work presents Sanvaad, a lightweight multimodal accessibility framework designed to support real time, two-way communication. For deaf users, Sanvaad includes an ISL recognition module built on MediaPipe landmarks. MediaPipe is chosen primarily for its efficiency and low computational load, enabling the system to run smoothly on edge devices without requiring dedicated hardware. Spoken input from a phone can also be translated into sign representations through a voice-to-sign component that maps detected speech to predefined phrases and produces corresponding GIFs or alphabet-based visualizations. For visually impaired users, the framework provides a screen free voice interface that integrates multilingual speech recognition, text summarization, and text-to-speech generation. These components work together through a Streamlit-based interface, making the system usable on both desktop and mobile environments. Overall, Sanvaad aims to offer a practical and accessible pathway for inclusive communication by combining lightweight computer vision and speech processing tools within a unified framework.

Related papers

• Kutti AI: A Voice-First, Offline-Capable Learning Companion with Real-Time Struggle Detection for Visually-Impaired Children [0.0]
  Kutti AI is a voice-first learning companion designed so that audio is the primary and sufficient interface.<n>We describe the architecture, the interaction flow, and the design decisions that prioritize accessibility.<n>Kutti AI illustrates how a small, carefully-engineered voice-first system can lower both accessibility and financial barriers to early education.
  arXiv  Detail & Related papers  (2026-07-24T15:03:41Z)
• VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility [0.5371337604556311]
  textbfVisionAssist is an open-source mobile application designed to improve independence by providing AI-powered visual assistance through a smartphone.<n>It generates spoken descriptions of captured images, allowing users to identify visual content such as food labels, documents, and everyday objects.<n>The application supports hands-free interaction through voice commands and delivers all feedback using text-to-speech synthesis.
  arXiv  Detail & Related papers  (2026-07-18T10:28:57Z)
• Covo-Audio Technical Report [61.09708870154148]
  Covo-Audio, a 7B-end LALM, directly processes continuous audio inputs and generates audio outputs within a single unified architecture.<n>Covo-Audio-Chat, a dialogue-oriented variant, demonstrates semantic strong spoken conversational abilities.
  arXiv  Detail & Related papers  (2026-02-10T14:31:11Z)
• VIBEVOICE-ASR Technical Report [95.57263110940973]
  VibeVoice-ASR addresses challenges of context fragmentation and multi-speaker complexity in long-form audio.<n>It supports over 50 languages, requires no explicit language setting, and handles code-switching within and across utterances.
  arXiv  Detail & Related papers  (2026-01-26T06:11:51Z)
• SeeingSounds: Learning Audio-to-Visual Alignment via Text [15.011814561603964]
  We introduce SeeingSounds, a framework for audio-to-image generation that leverages the interplay between audio, language, and vision.<n>Our method performs dual alignment: audio is projected into a semantic language space via a frozen language encoder, and, contextually grounded into the visual domain using a vision-language model.<n>This approach, inspired by cognitive neuroscience, reflects the natural cross-modal associations observed in human perception.
  arXiv  Detail & Related papers  (2025-10-10T18:42:50Z)
• Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio [52.859261069569165]
  We propose the first unified framework capable of handling diverse combinations of sign language, lip movements, and audio for spoken-language text generation.<n>We focus on three main objectives: (i) designing a unified, modality-agnostic architecture capable of effectively processing heterogeneous inputs; (ii) exploring the underexamined synergy among modalities, particularly the role of lip movements as non-manual cues in sign language comprehension; and (iii) achieving performance on par with or better than state-of-the-art models specialized for individual tasks.
  arXiv  Detail & Related papers  (2025-08-28T06:51:42Z)
• Real-Time Sign Language Gestures to Speech Transcription using Deep Learning [0.0]
  This project introduces a real-time assistive technology solution that leverages advanced deep learning techniques to translate sign language gestures into textual and audible speech.<n>By employing convolution neural networks (CNN) trained on the Sign Language MNIST dataset, the system accurately classifies hand gestures captured live via webcam.
  arXiv  Detail & Related papers  (2025-08-18T08:25:18Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment [15.899112804399193]
  We present textbfTESU-LLM, a novel framework that enables training speech-capable language models using only text data.<n>Our key insight is to leverage a unified encoder that maps semantically equivalent text and speech inputs to a shared latent space.<n>Despite being trained exclusively on text, TESU-LLM achieves strong performance on various speech-related benchmarks.
  arXiv  Detail & Related papers  (2025-06-01T09:27:55Z)
• Your voice is your voice: Supporting Self-expression through Speech Generation and LLMs in Augmented and Alternative Communication [9.812902134556971]
  Speak Ease is an augmentative and alternative communication system to support users' expressivity.<n>System integrates multimodal input, including text, voice, and contextual cues, with large language models.
  arXiv  Detail & Related papers  (2025-03-21T18:50:05Z)
• MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition [51.412413996510814]
  We propose MixSpeech, a cross-modality self-learning framework that utilizes audio speech to regularize the training of visual speech tasks. MixSpeech enhances speech translation in noisy environments, improving BLEU scores for four languages on AVMuST-TED by +1.4 to +4.2.
  arXiv  Detail & Related papers  (2023-03-09T14:58:29Z)
• VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning [119.49605266839053]
  We propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model) The proposed VATLM employs a unified backbone network to model the modality-independent information. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens.
  arXiv  Detail & Related papers  (2022-11-21T09:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.