X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

• URL: http://arxiv.org/abs/2609.11412v1
• Date: Thu, 10 Sep 2026 11:43:17 GMT
• Title: X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
• Abstract summary: X-AuT is a progressive framework that selects layer combinations through short behavioral probes.<n>It restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning.<n>On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%.
• Score: 34.99298958122614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18$\rightarrow$14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut

Related papers

• Stride-k Subsampling: Train-Free Audio Token Reduction for Whisper [4.214795904343116]
  We propose stride-k subsampling, a deterministic indexing operation that retains every k-th token after the convolutional stem or encoder transformer.<n>Applying stride-2 at both positions cuts audio tokens by 75% and total GFLOPs by 52-58%, with small WER costs on most ASR benchmarks and larger costs on harder ones.<n>The same configuration extends to three Whisper-based SpeechLMs, yielding modest accuracy drops on stronger baselines and larger drops on weaker ones, while reducing end-to-end latency by 19.6-27.4%.
  arXiv  Detail & Related papers  (2026-08-31T15:04:29Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Voice Memory for Agentic Speech Recognition [66.69623133635868]
  We present Voice Memory, a inference-only scheme for agentic speech recognition.<n>Asynchronously, a score-gated example revises that file through bounded edits.<n>Across ten HyPoradise domains with an open corrector, Voice Memory, weighted word error rate from 8.36% to 7.52%.
  arXiv  Detail & Related papers  (2026-07-29T02:42:56Z)
• Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers [76.15132587294862]
  Wide-In, Narrow-Out (WINO) is a training-free decoding algorithm that enables revokable parallel generation.<n>WINO+ injects the verified denoising trajectories produced by WINO into model parameters, aligning training with efficient inference.<n>Experiments on LLaDA and MMaDA show that WINO improves both quality and efficiency, while WINO+ further strengthens this progression.
  arXiv  Detail & Related papers  (2026-05-16T11:27:40Z)
• Pushing the Limits of On-Device Streaming ASR: A Compact, High-Accuracy English Model for Low-Latency Inference [5.685908474026849]
  High-quality automatic speech recognition (ASR) on edge devices requires models that jointly optimize accuracy, latency, and memory footprint while operating entirely on CPU without GPU acceleration.<n>We conduct a systematic empirical study of state-of-the-art ASR architectures, encompassing encoder-decoder, transducer, and LLM-based paradigms, evaluated across batch, chunked, and streaming inference modes.<n>Our recommended configuration, the int4 k-quant variant, achieves 8.20% average streaming WER across eight standard benchmarks, running comfortably faster than real-time on CPU with 0.56 s algorithmic latency.
  arXiv  Detail & Related papers  (2026-04-16T00:04:32Z)
• Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes [3.305265383862785]
  We present an empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder finetuning with companion representation probes.<n>In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9% +/- average forgetting, whereas standard LoRA (r, query/value modules) yields 0.6% +/-1.4%.
  arXiv  Detail & Related papers  (2026-03-29T14:14:36Z)
• dVoting: Fast Voting for dLLMs [71.572316901001]
  Diffusion Large Language Models (dLLMs) represent a new paradigm beyond autoregressive modeling.<n>dLLMs can generate tokens at arbitrary positions in parallel, endowing them with significant potential for parallel test-time scaling.<n>We introduce dVoting, a fast voting technique that boosts reasoning capability without training, with only an acceptable extra computational overhead.
  arXiv  Detail & Related papers  (2026-02-12T16:35:05Z)
• Adding Connectionist Temporal Summarization into Conformer to Improve Its Decoder Efficiency For Speech Recognition [22.61761934996406]
  We propose a novel connectionist temporal summarization (CTS) method that reduces the number of frames required for the attention decoder. With a beamwidth of 4, the LibriSpeech's decoding budget can be reduced by up to 20%. The word error rate (WER) is reduced by 6% relative at the beam width of 1 and by 3% relative at the beam width of 4.
  arXiv  Detail & Related papers  (2022-04-08T07:24:00Z)
• Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection [15.525314212209562]
  We propose three latency reduction techniques for chunk-based incremental inference. We show that our approach is also applicable to low-latency speech translation.
  arXiv  Detail & Related papers  (2020-05-22T13:42:54Z)
• Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition [66.47000813920619]
  We propose a non-autoregressive end-to-end speech recognition system called LASO. Because of the non-autoregressive property, LASO predicts a textual token in the sequence without the dependence on other tokens. We conduct experiments on publicly available Chinese dataset AISHELL-1.
  arXiv  Detail & Related papers  (2020-05-11T04:45:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.