Kutti AI: A Voice-First, Offline-Capable Learning Companion with Real-Time Struggle Detection for Visually-Impaired Children

• URL: http://arxiv.org/abs/2607.22377v1
• Date: Fri, 24 Jul 2026 15:03:41 GMT
• Title: Kutti AI: A Voice-First, Offline-Capable Learning Companion with Real-Time Struggle Detection for Visually-Impaired Children
• Abstract summary: Kutti AI is a voice-first learning companion designed so that audio is the primary and sufficient interface.<n>We describe the architecture, the interaction flow, and the design decisions that prioritize accessibility.<n>Kutti AI illustrates how a small, carefully-engineered voice-first system can lower both accessibility and financial barriers to early education.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most educational technology for children is built around visual interfaces, which excludes the many children worldwide who live with visual impairment -- an estimated 1.4 million children are blind and many more have low vision. We present Kutti AI, a voice-first learning companion designed so that audio is the primary and sufficient interface: children learn curriculum concepts through spoken conversation, respond by speaking, and receive spoken feedback, with no reliance on visual elements. The system contributes three practical mechanisms for accessible, adaptive learning on commodity mobile hardware: (1) a multi-signal struggle-detection engine that combines response-latency analysis, wrong-attempt tracking, and keyword-based hesitation detection to decide, in real time, when to offer hints or simplify a question; (2) a multi-layered cross-language answer-matching pipeline that combines language-aware translation/transliteration, Levenshtein-based fuzzy matching, and text normalization so that children are not penalized for code-switching or pronunciation variation; and (3) an offline-first speech pipeline using an on-device automatic speech recognition (ASR) model, enabling use in low-connectivity settings common in underserved communities. We describe the architecture, the interaction flow, and the design decisions that prioritize accessibility, and we report qualitative observations from a hackathon prototype supporting English and Tamil. We discuss lessons learned and outline a path toward formal evaluation with target users. Kutti AI illustrates how a small, carefully-engineered voice-first system can lower both accessibility and financial barriers to early education.

Related papers

• KidSpeak: A General Multi-purpose LLM for Kids' Speech Recognition and Screening [29.54910094759367]
  KidSpeak is a speech-enhanced Foundation Model capable of both generative and discriminative tasks specifically tailored to children's speech patterns.<n>We propose the Flexible and Automatic Speech Aligner (FASA) and leverage the method to construct high quality datasets for training and evaluation.<n>This novel alignment tool significantly improves the quality of aligned children's speech from noisy data, enhancing data quality by 13.6x compared to human annotations.
  arXiv  Detail & Related papers  (2025-12-01T00:19:37Z)
• ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction [88.41471266579333]
  We propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models.<n> Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones show significant improvements.
  arXiv  Detail & Related papers  (2025-11-09T08:50:11Z)
• Towards Inclusive Communication: A Unified Framework for Generating Spoken Language from Sign, Lip, and Audio [52.859261069569165]
  We propose the first unified framework capable of handling diverse combinations of sign language, lip movements, and audio for spoken-language text generation.<n>We focus on three main objectives: (i) designing a unified, modality-agnostic architecture capable of effectively processing heterogeneous inputs; (ii) exploring the underexamined synergy among modalities, particularly the role of lip movements as non-manual cues in sign language comprehension; and (iii) achieving performance on par with or better than state-of-the-art models specialized for individual tasks.
  arXiv  Detail & Related papers  (2025-08-28T06:51:42Z)
• Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models [49.22720751953838]
  We propose a method for training language models in an interactive setting inspired by child language acquisition.<n>In our setting, a speaker attempts to communicate some information to a listener in a single-turn dialogue and receives a reward if communicative success is achieved.
  arXiv  Detail & Related papers  (2025-05-09T11:48:36Z)
• Developmental Predictive Coding Model for Early Infancy Mono and Bilingual Vocal Continual Learning [69.8008228833895]
  We propose a small-sized generative neural network equipped with a continual learning mechanism.<n>Our model prioritizes interpretability and demonstrates the advantages of online learning.
  arXiv  Detail & Related papers  (2024-12-23T10:23:47Z)
• CLIP-VAD: Exploiting Vision-Language Models for Voice Activity Detection [2.110168344647122]
  Voice Activity Detection (VAD) is the process of automatically determining whether a person is speaking and identifying the timing of their speech. We introduce a novel approach leveraging Contrastive Language-Image Pretraining (CLIP) models. Our approach outperforms several audio-visual methods despite its simplicity, and without requiring pre-training on extensive audio-visual datasets.
  arXiv  Detail & Related papers  (2024-10-18T14:43:34Z)
• Language-Guided Audio-Visual Source Separation via Trimodal Consistency [64.0580750128049]
  A key challenge in this task is learning to associate the linguistic description of a sound-emitting object to its visual features and the corresponding components of the audio waveform. We adapt off-the-shelf vision-language foundation models to provide pseudo-target supervision via two novel loss functions. We demonstrate the effectiveness of our self-supervised approach on three audio-visual separation datasets.
  arXiv  Detail & Related papers  (2023-03-28T22:45:40Z)
• Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech [26.076534338576234]
  Learning to understand grounded language, which connects natural language to percepts, is a critical research area. In this work we demonstrate the feasibility of performing grounded language acquisition on paired visual percepts and raw speech inputs.
  arXiv  Detail & Related papers  (2021-12-27T16:12:30Z)
• Hierarchical Summarization for Longform Spoken Dialog [1.995792341399967]
  Despite the pervasiveness of spoken dialog, automated speech understanding and quality information extraction remains markedly poor. Compared to understanding text, auditory communication poses many additional challenges such as speaker disfluencies, informal prose styles, and lack of structure. We propose a two stage ASR and text summarization pipeline and propose a set of semantic segmentation and merging algorithms to resolve these speech modeling challenges.
  arXiv  Detail & Related papers  (2021-08-21T23:31:31Z)
• An Adversarial Learning based Multi-Step Spoken Language Understanding System through Human-Computer Interaction [70.25183730482915]
  We introduce a novel multi-step spoken language understanding system based on adversarial learning. We demonstrate that the new system can improve parsing performance by at least $2.5%$ in terms of F1.
  arXiv  Detail & Related papers  (2021-06-06T03:46:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.