CodeVaani: A Multilingual, Voice-Based Code Learning Assistant

• URL: http://arxiv.org/abs/2511.20654v1
• Date: Wed, 08 Oct 2025 17:15:59 GMT
• Title: CodeVaani: A Multilingual, Voice-Based Code Learning Assistant
• Abstract summary: We present CodeVaani, a multilingual speech-driven assistant for understanding code built into Bodhitree.<n>It is a voice-enabled assistant that helps learners explore programming concepts in their native languages.<n>In a study with 28 beginner programmers, CodeVaani achieved 75% response accuracy, with over 80% of participants rating the experience positively.
• Score: 13.984753319694763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Programming education often assumes English proficiency and text-based interaction, creating barriers for students from multilingual regions such as India. We present CodeVaani, a multilingual speech-driven assistant for understanding code, built into Bodhitree [1], a Learning Management System developed at IIT Bombay. It is a voice-enabled assistant that helps learners explore programming concepts in their native languages. The system integrates Indic ASR, a codeaware transcription refinement module, and a code model for generating relevant answers. Responses are provided in both text and audio for natural interaction. In a study with 28 beginner programmers, CodeVaani achieved 75% response accuracy, with over 80% of participants rating the experience positively. Compared to classroom assistance, our framework offers ondemand availability, scalability to support many learners, and multilingual support that lowers the entry barrier for students with limited English proficiency. The demo will illustrate these capabilities and highlight how voice-based AI systems can make programming education more inclusive. Supplementary artifacts and demo video are also made available.

Related papers

• Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding [16.695144374746622]
  We develop a speech-driven framework for code understanding that accepts spoken queries in a user native language.<n>We systematically characterize how transcription errors impact downstream task performance.<n>Our findings underscore the need for code-sensitive adaptations in speech interfaces.
  arXiv  Detail & Related papers  (2026-01-20T14:34:40Z)
• Omnilingual ASR: Open-Source Multilingual Speech Recognition for 1600+ Languages [76.14451035425229]
  We introduce Omnilingual ASR, a large-scale automatic speech recognition system.<n>It scales self-supervised pre-training to 7B parameters to learn robust speech representations.<n>It expands coverage to over 1,600 languages, including over 500 never before served by ASR.
  arXiv  Detail & Related papers  (2025-11-12T19:48:09Z)
• GenAI Voice Mode in Programming Education [0.0]
  Real-time voice interfaces using multimodal Generative AI (GenAI) can potentially address the accessibility needs of novice programmers with disabilities.<n>This paper analyzes audio dialogues from nine 9th-grade students using a voice-enabled tutor in an authentic classroom setting while learning Python.
  arXiv  Detail & Related papers  (2025-09-12T15:25:08Z)
• Breaking the Programming Language Barrier: Multilingual Prompting to Empower Non-Native English Learners [3.1550561074143597]
  Non-native English speakers (NNES) face multiple barriers to learning programming.<n>Advances in generative AI (GenAI) have the potential to break down these barriers.<n>In this paper, we provide the first exploration of NNES students prompting in their native languages to generate code.
  arXiv  Detail & Related papers  (2024-12-17T11:06:02Z)
• CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs [14.866602418099518]
  LLM-powered programming assistant CodeAid delivers technically correct responses without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student's incorrect code with fix suggestions.
  arXiv  Detail & Related papers  (2024-01-20T20:14:42Z)
• Teacher Perception of Automatically Extracted Grammar Concepts for L2 Language Learning [66.79173000135717]
  We apply this work to teaching two Indian languages, Kannada and Marathi, which do not have well-developed resources for second language learning. We extract descriptions from a natural text corpus that answer questions about morphosyntax (learning of word order, agreement, case marking, or word formation) and semantics (learning of vocabulary). We enlist the help of language educators from schools in North America to perform a manual evaluation, who find the materials have potential to be used for their lesson preparation and learner evaluation.
  arXiv  Detail & Related papers  (2023-10-27T18:17:29Z)
• Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge [57.38948190611797]
  This paper proposes a novel lip reading framework, especially for low-resource languages. Since low-resource languages do not have enough video-text paired data to train the model, it is regarded as challenging to develop lip reading models for low-resource languages.
  arXiv  Detail & Related papers  (2023-08-18T05:19:03Z)
• AudioPaLM: A Large Language Model That Can Speak and Listen [79.44757696533709]
  We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models. It can process and generate text and speech with applications including speech recognition and speech-to-speech translation.
  arXiv  Detail & Related papers  (2023-06-22T14:37:54Z)
• LAE: Language-Aware Encoder for Monolingual and Multilingual ASR [87.74794847245536]
  A novel language-aware encoder (LAE) architecture is proposed to handle both situations by disentangling language-specific information. Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level.
  arXiv  Detail & Related papers  (2022-06-05T04:03:12Z)
• Code Switched and Code Mixed Speech Recognition for Indic languages [0.0]
  Training multilingual automatic speech recognition (ASR) systems is challenging because acoustic and lexical information is typically language specific. We compare the performance of end to end multilingual speech recognition system to the performance of monolingual models conditioned on language identification (LID) We also propose a similar technique to solve the Code Switched problem and achieve a WER of 21.77 and 28.27 over Hindi-English and Bengali-English respectively.
  arXiv  Detail & Related papers  (2022-03-30T18:09:28Z)
• Multilingual and code-switching ASR challenges for low resource Indian languages [59.2906853285309]
  We focus on building multilingual and code-switching ASR systems through two different subtasks related to a total of seven Indian languages. We provide a total of 600 hours of transcribed speech data, comprising train and test sets, in these languages. We also provide a baseline recipe for both the tasks with a WER of 30.73% and 32.45% on the test sets of multilingual and code-switching subtasks, respectively.
  arXiv  Detail & Related papers  (2021-04-01T03:37:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.