Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs

• URL: http://arxiv.org/abs/2604.08003v1
• Date: Thu, 09 Apr 2026 09:07:52 GMT
• Title: Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs
• Abstract summary: We introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM.<n>We propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and robustness hallucination.<n> Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters.
• Score: 17.167595029948576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrating large language models (LLMs) into automatic speech recognition (ASR) has become a dominant paradigm. Although recent LLM-based ASR models have shown promising performance on public benchmarks, it remains challenging to balance recognition quality with latency and overhead, while hallucinations further limit real-world deployment. In this study, we revisit LLM-based ASR from an entropy allocation perspective and introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM. To remedy entropy-allocation inefficiencies in prevailing approaches, we propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and hallucination robustness. Specifically, we redesign the pretraining strategy to alleviate the speech-text modality gap, and further introduce an iterative asynchronous SFT stage between alignment and joint SFT to preserve functional decoupling and constrain encoder representation drift. Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters, while also effectively mitigating hallucinations through our decoupling-oriented design.

Related papers

• Listen to the Latents: Self-Correcting Speech Recognition in Large Audio Language Models Through Hidden-State Interactions [120.64235456332916]
  Speech recognition systems increasingly integrate large language models (LLMs) to leverage their semantic knowledge.<n>We refine warm-adapted LLM-based ASR models by leveraging their own pre-adaptation base LLMs.<n>We propose Hybrid Search, a targeted correction strategy motivated by two observations.
  arXiv  Detail & Related papers  (2026-08-31T15:46:48Z)
• NIM4-ASR: Towards Efficient, Robust, and Customizable Real-Time LLM-Based ASR [22.527587147157462]
  We present NIM4-ASR, a production-oriented LLM-based ASR framework optimized for both efficiency and robustness.<n>We reformulate the pre-training architecture and objective to mitigate the modality gap and improve parameter efficiency.<n>We additionally incorporate a suite of production-oriented optimizations, including robustness under noisy and silent conditions.
  arXiv  Detail & Related papers  (2026-04-20T11:21:06Z)
• CoLLM: A Unified Framework for Co-execution of LLMs Federated Fine-tuning and Inference [33.14453252489961]
  CoLLM is a system that unifies FL PEFT and inference on shared edge replicas and model parameters.<n>It consistently outperforms state-of-the-art LLM systems, achieving up to 3x higher goodput.
  arXiv  Detail & Related papers  (2026-03-31T09:49:47Z)
• Lightweight Adaptation for LLM-based Technical Service Agent: Latent Logic Augmentation and Robust Noise Reduction [15.613490987782264]
  Adapting Large Language Models in complex technical service domains is constrained by the absence of explicit cognitive chains in human demonstrations.<n>We introduce Planning-Aware Trajectory Modeling and Decision Reasoning Augmentation to bridge the gap between surface-level supervision and latent decision logic.<n>We construct a Multiple Ground Truths dataset through a dual-filtering method to reduce the noise by validating diverse responses.
  arXiv  Detail & Related papers  (2026-03-18T05:01:17Z)
• Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale [12.626090218930578]
  Single-codebook text-to-speech models often exhibit unstable prosody, speaker drift, and degraded naturalness.<n>We propose a multi-reward Group Relative Policy Optimization framework that directly optimize the token generation policy of single-codebook TTS LLMs.<n>We show that the proposed method consistently enhances prosodic stability, speaker similarity, and overall speech naturalness in single-codebook TTS LLMs.
  arXiv  Detail & Related papers  (2025-11-26T10:50:17Z)
• Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models [64.92045568376705]
  Coherent Contextual Decoding (CCD) is a novel inference framework built upon two core innovations.<n>CCD employs a trajectory rectification mechanism that leverages historical context to enhance sequence coherence.<n>Instead of rigid allocations based on diffusion steps, we introduce an adaptive sampling strategy that dynamically adjusts the unmasking budget for each step.
  arXiv  Detail & Related papers  (2025-11-26T09:49:48Z)
• Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models [34.15708407614003]
  Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities.<n>We present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation.<n> Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-10T16:03:44Z)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding [70.01564145836129]
  This paper conceptualizes the decoding process as analogous to the ranking stage in recommendation pipelines.<n>Motivated by this insight, we propose Language Ranker, a novel framework that introduces a lightweight module to rerank candidate responses.<n> Experiments show that Language Ranker achieves performance comparable to large-scale reward models, while requiring only 0.5M additional parameters.
  arXiv  Detail & Related papers  (2025-10-23T17:56:46Z)
• Transformer-based Scalable Beamforming Optimization via Deep Residual Learning [12.79709425087431]
  unsupervised deep learning framework for downlink beamforming in large-scale MU-MISO channels.<n>Model is trained offline, allowing real-time inference through lightweight feedforward computations in dynamic communication environments.
  arXiv  Detail & Related papers  (2025-10-15T01:43:51Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• A Novel Speech Intelligibility Enhancement Model based on CanonicalCorrelation and Deep Learning [12.913738983870621]
  We present a canonical correlation based short-time objective intelligibility (CC-STOI) cost function to train a fully convolutional neural network (FCN) model. We show that our CC-STOI based speech enhancement framework outperforms state-of-the-art DL models trained with conventional distance-based and STOI-based loss functions.
  arXiv  Detail & Related papers  (2022-02-11T16:48:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.