Data-Efficient On-Policy Distillation for Automatic Speech Recognition

• URL: http://arxiv.org/abs/2605.28139v1
• Date: Wed, 27 May 2026 08:22:47 GMT
• Title: Data-Efficient On-Policy Distillation for Automatic Speech Recognition
• Abstract summary: We study Ark-ASR, a 0.6B- parameter audio-conditioned language model trained with 100k hours of speech.<n>We examine whether a strong Qwen-ASR teacher can transfer additional recognition capability through on-policy distillation.
• Score: 9.538470992436883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building competitive automatic speech recognition (ASR) models usually requires large-scale au- dio supervision, which makes reproduction and specialization expensive. We study Ark-ASR, a 0.6B- parameter audio-conditioned language model trained with 100k hours of speech, and examine whether a strong Qwen-ASR teacher can transfer additional recognition capability through on-policy distillation. Across Mandarin and English ASR benchmarks, the proposed training recipe consistently improves over supervised fine-tuning alone and outperforms the same-scale Qwen3-ASR-0.6B baseline on four of five evaluation sets. This is achieved with only 100k hours of speech, compared with the 20M hours of super- vised audio reported for the Qwen3-Omni AuT encoder. The larger Qwen3-ASR-1.7B remains stronger, but the results show that teacher-guided on-policy training can substantially close the gap for compact ASR models under a much smaller audio budget. A support-overlap diagnostic further suggests that the teacher-data stage improves local student-teacher compatibility, matching recent analyses of when on-policy distillation is effective.

Related papers

• Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision [50.61441331643804]
  Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training.<n>Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations.<n>We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations.
  arXiv  Detail & Related papers  (2026-04-13T19:46:55Z)
• Self-Distilled RLVR [57.37526213765131]
  We show that learning signals solely derived from the privileged teacher result in severe information leakage and unstable long-term training.<n>We propose textbfRLSD (textbfRLVR with textbfSelf-textbfDistillation)<n>This enables RLSD to simultaneously harness the strengths of both RLVR and OPSD, achieving a higher convergence ceiling and superior training stability.
  arXiv  Detail & Related papers  (2026-04-03T15:50:07Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• Seewo's Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models [4.917936997225074]
  Seewo's systems for both tracks of the Multilingual Conversational Speech Language Model Challenge (MLC-SLM)<n>We introduce a multi-stage training pipeline that explicitly enhances reasoning and self-correction in speech language models for ASR.
  arXiv  Detail & Related papers  (2025-06-16T09:42:05Z)
• End-to-End Transformer-based Automatic Speech Recognition for Northern Kurdish: A Pioneering Approach [1.3689715712707342]
  This paper introduces a study exploring the effectiveness of Whisper, a pre-trained ASR model, for Northern Kurdish (Kurmanji) an under-resourced language spoken in the Middle East. Using a Northern Kurdish fine-tuning speech corpus containing approximately 68 hours of validated transcribed data, our experiments demonstrate that the additional module fine-tuning strategy significantly improves ASR accuracy.
  arXiv  Detail & Related papers  (2024-10-19T11:46:30Z)
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition [50.93943755401025]
  We propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses.
  arXiv  Detail & Related papers  (2023-01-19T02:37:56Z)
• Analyzing And Improving Neural Speaker Embeddings for ASR [54.30093015525726]
  We present our efforts w.r.t integrating neural speaker embeddings into a conformer based hybrid HMM ASR system. Our best Conformer-based hybrid ASR system with speaker embeddings achieves 9.0% WER on Hub5'00 and Hub5'01 with training on SWB 300h.
  arXiv  Detail & Related papers  (2023-01-11T16:56:03Z)
• Automatic Severity Classification of Dysarthric speech by using Self-supervised Model with Multi-task Learning [4.947423926765435]
  We propose a novel automatic severity assessment method for dysarthric speech using the self-supervised model in conjunction with multi-task learning. Wav2vec 2.0 XLS-R is trained for two different tasks: severity classification and auxiliary automatic speech recognition (ASR) Our model outperforms the traditional baseline methods, with a relative percentage increase of 1.25% for F1-score.
  arXiv  Detail & Related papers  (2022-10-27T12:48:10Z)
• Scenario Aware Speech Recognition: Advancements for Apollo Fearless Steps & CHiME-4 Corpora [70.46867541361982]
  We consider a general non-semantic speech representation, which is trained with a self-supervised criteria based on triplet loss called TRILL. We observe +5.42% and +3.18% relative WER improvement for the development and evaluation sets of Fearless Steps.
  arXiv  Detail & Related papers  (2021-09-23T00:43:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.