Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognition

• URL: http://arxiv.org/abs/2512.17247v1
• Date: Fri, 19 Dec 2025 05:26:50 GMT
• Title: Incorporating Error Level Noise Embedding for Improving LLM-Assisted Robustness in Persian Speech Recognition
• Authors: Zahra Rahmani, Hossein Sameti,
• Abstract summary: This study presents a robust noise-sensitive ASR error correction framework that combines multiple hypotheses and noise-aware modeling.<n>Using noisy Persian speech, we generate 5-best hypotheses from a modified Whisper-large decoder.<n> Experimental results demonstrate that the ELN-conditioned model achieves substantial reductions in Word Error Rate (WER)
• Score: 1.3607388598209322
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Automatic Speech Recognition (ASR) systems suffer significant performance degradation in noisy environments, a challenge that is especially severe for low-resource languages such as Persian. Even state-of-the-art models such as Whisper struggle to maintain accuracy under varying signal-to-noise ratios (SNRs). This study presents a robust noise-sensitive ASR error correction framework that combines multiple hypotheses and noise-aware modeling. Using noisy Persian speech, we generate 5-best hypotheses from a modified Whisper-large decoder. Error Level Noise (ELN) is introduced as a representation that captures semantic- and token-level disagreement across hypotheses, quantifying the linguistic distortions caused by noise. ELN thus provides a direct measure of noise-induced uncertainty, enabling the LLM to reason about the reliability of each hypothesis during correction. Three models are evaluated: (1) a base LLaMA-2-7B model without fine-tuning, (2) a fine-tuned variant trained on text-only hypotheses, and (3) a noise-conditioned model integrating ELN embeddings at both sentence and word levels. Experimental results demonstrate that the ELN-conditioned model achieves substantial reductions in Word Error Rate (WER). Specifically, on the challenging Mixed Noise test set, the proposed Fine-tuned + ELN (Ours) model reduces the WER from a baseline of 31.10\% (Raw Whisper) to 24.84\%, significantly surpassing the Fine-tuned (No ELN) text-only baseline of 30.79\%, whereas the original LLaMA-2-7B model increased the WER to 64.58\%, demonstrating that it is unable to correct Persian errors on its own. This confirms the effectiveness of combining multiple hypotheses with noise-aware embeddings for robust Persian ASR in noisy real-world scenarios.

Related papers

• Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses [71.34350093068473]
  This paper introduces a new paradigm for generative error correction (GER) framework in audio-visual speech recognition (AVSR)<n>Our framework, DualHyp, empowers a large language model (LLM) to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition (VSR) models.<n>Our framework attains up to 57.7% error rate gain on the LRS2 benchmark over standard ASR baseline, contrary to single-stream GER approaches that achieve only 10% gain.
  arXiv  Detail & Related papers  (2025-10-15T08:27:16Z)
• Mitigating the Noise Shift for Denoising Generative Models via Noise Awareness Guidance [54.88271057438763]
  Noise Awareness Guidance (NAG) is a correction method that explicitly steers sampling trajectories to remain consistent with the pre-defined noise schedule.<n>NAG consistently mitigates noise shift and substantially improves the generation quality of mainstream diffusion models.
  arXiv  Detail & Related papers  (2025-10-14T13:31:34Z)
• Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models [1.0579965347526206]
  Large language models (LLMs) often produce inaccurate or misleading content-hallucinations.<n>Noise-Augmented Fine-Tuning (NoiseFiT) is a novel framework that leverages adaptive noise injection to enhance model robustness.<n>NoiseFiT selectively perturbs layers identified as either high-SNR (more robust) or low-SNR (potentially under-regularized) using a dynamically scaled Gaussian noise.
  arXiv  Detail & Related papers  (2025-04-04T09:27:19Z)
• Do we really have to filter out random noise in pre-training data for language models? [38.30911856420568]
  Pre-training text data curated from the Internet inevitably contains random noise caused by decoding errors or unregulated web content.<n>We show that the resulting increase in the loss of next-token prediction (NTP) was significantly lower than the proportion of random noise even when the model was scaled up to 2.7B.<n>We introduce a novel plug-and-play Local Gradient Matching loss, which explicitly enhances the denoising capability of the downstream task head by aligning the gradient of normal and perturbed features without requiring knowledge of the model's parameters.
  arXiv  Detail & Related papers  (2025-02-10T16:01:55Z)
• DENOASR: Debiasing ASRs through Selective Denoising [5.544079217915537]
  We introduce a novel framework DENOASR, which is a selective denoising technique to reduce the disparity in the word error rates between the two gender groups. We find that a combination of two popular speech denoising techniques, viz. DEMUCS and LE, can be effectively used to mitigate ASR disparity without compromising their overall performance.
  arXiv  Detail & Related papers  (2024-10-22T05:39:24Z)
• Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing [6.86204821852287]
  Randomized smoothing is the primary certified robustness method for accessing the robustness of deep learning models to adversarial perturbations in the l2-norm. A notable constraint limiting widespread adoption is the necessity to retrain base models entirely from scratch to attain a robust version. This is because the base model fails to learn the noise-augmented data distribution to give an accurate vote. Inspired by recent large model training procedures, we explore an alternative way named PEFTSmoothing to adapt the base model to learn the noise-augmented data.
  arXiv  Detail & Related papers  (2024-04-08T09:38:22Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)
• Large Language Models are Efficient Learners of Noise-Robust Speech Recognition [65.95847272465124]
  Recent advances in large language models (LLMs) have promoted generative error correction (GER) for automatic speech recognition (ASR) In this work, we extend the benchmark to noisy conditions and investigate if we can teach LLMs to perform denoising for GER. Experiments on various latest LLMs demonstrate our approach achieves a new breakthrough with up to 53.9% correction improvement in terms of word error rate.
  arXiv  Detail & Related papers  (2024-01-19T01:29:27Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.