Back to Basics: Revisiting ASR in the Age of Voice Agents

• URL: http://arxiv.org/abs/2603.25727v1
• Date: Thu, 26 Mar 2026 17:59:03 GMT
• Title: Back to Basics: Revisiting ASR in the Age of Voice Agents
• Abstract summary: We introduce WildASR, a diagnostic benchmark sourced entirely from real human speech.<n>We find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions.<n>Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems.
• Score: 25.372625280850695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic speech recognition (ASR) systems have achieved near-human accuracy on curated benchmarks, yet still fail in real-world voice agents under conditions that current evaluations do not systematically cover. Without diagnostic tools that isolate specific failure factors, practitioners cannot anticipate which conditions, in which languages, will cause what degree of degradation. We introduce WildASR, a multilingual (four-language) diagnostic benchmark sourced entirely from real human speech that factorizes ASR robustness along three axes: environmental degradation, demographic shift, and linguistic diversity. Evaluating seven widely used ASR systems, we find severe and uneven performance degradation, and model robustness does not transfer across languages or conditions. Critically, models often hallucinate plausible but unspoken content under partial or degraded inputs, creating concrete safety risks for downstream agent behavior. Our results demonstrate that targeted, factor-isolated evaluation is essential for understanding and improving ASR reliability in production systems. Besides the benchmark itself, we also present three analytical tools that practitioners can use to guide deployment decisions.

Related papers

• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• The Necessity of a Unified Framework for LLM-Based Agent Evaluation [46.631678638677386]
  General-purpose agents have seen fundamental advancements.<n> evaluating these agents presents unique challenges that distinguish them from static QA benchmarks.<n>We propose that a unified evaluation framework is essential for the rigorous advancement of agent evaluation.
  arXiv  Detail & Related papers  (2026-02-03T08:18:37Z)
• Lost in the Noise: How Reasoning Models Fail with Contextual Distractors [57.31788955167306]
  Recent advances in reasoning models and agentic AI systems have led to an increased reliance on diverse external information.<n>We introduce NoisyBench, a comprehensive benchmark that systematically evaluates model robustness across 11 datasets in RAG, reasoning, alignment, and tool-use tasks.<n>Our evaluation reveals a catastrophic performance drop of up to 80% in state-of-the-art models when faced with contextual distractors.
  arXiv  Detail & Related papers  (2026-01-12T05:43:51Z)
• LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models [49.92148175114169]
  We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions.<n>Models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states.<n>Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely.
  arXiv  Detail & Related papers  (2025-10-15T14:51:36Z)
• Data-Efficient ASR Personalization for Non-Normative Speech Using an Uncertainty-Based Phoneme Difficulty Score for Guided Sampling [2.0128859854921743]
  This work introduces a data-efficient personalization method that quantifies phoneme-level uncertainty to guide fine-tuning.<n>We leverage Monte Carlo Dropout to estimate which phonemes a model finds most difficult.<n>Our results show that this clinically-validated, uncertainty-guided sampling significantly improves ASR accuracy, delivering a practical framework for personalized and inclusive ASR.
  arXiv  Detail & Related papers  (2025-09-23T12:54:30Z)
• SVeritas: Benchmark for Robust Speaker Verification under Diverse Conditions [54.34001921326444]
  Speaker verification (SV) models are increasingly integrated into security, personalization, and access control systems.<n>Existing benchmarks evaluate only subsets of these conditions, missing others entirely.<n>We introduce SVeritas, a comprehensive Speaker Verification tasks benchmark suite, assessing SV systems under stressors like recording duration, spontaneity, content, noise, microphone distance, reverberation, channel mismatches, audio bandwidth, codecs, speaker age, and susceptibility to spoofing and adversarial attacks.
  arXiv  Detail & Related papers  (2025-09-21T14:11:16Z)
• Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia [0.5242869847419834]
  Audits often adhere to a single method of text standardization during data pre-processing.<n>Third, audits often rely on a single gold-standard metric -- the Word Error Rate.<n>We propose a more holistic auditing framework that accounts for these three pitfalls.
  arXiv  Detail & Related papers  (2025-06-10T14:34:36Z)
• PSRB: A Comprehensive Benchmark for Evaluating Persian ASR Systems [0.0]
  This paper introduces Persian Speech Recognition Benchmark(PSRB), a comprehensive benchmark designed to address this gap by incorporating diverse linguistic and acoustic conditions.<n>We evaluate ten ASR systems, including state-of-the-art commercial and open-source models, to examine performance variations and inherent biases.<n>Our findings indicate that while ASR models generally perform well on standard Persian, they struggle with regional accents, children's speech, and specific linguistic challenges.
  arXiv  Detail & Related papers  (2025-05-27T14:14:55Z)
• Toward Practical Automatic Speech Recognition and Post-Processing: a Call for Explainable Error Benchmark Guideline [12.197453599489963]
  We propose the development of an Error Explainable Benchmark (EEB) dataset. This dataset, while considering both speech- and text-level, enables a granular understanding of the model's shortcomings. Our proposition provides a structured pathway for a more real-world-centric' evaluation, allowing for the detection and rectification of nuanced system weaknesses.
  arXiv  Detail & Related papers  (2024-01-26T03:42:45Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.