Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild

• URL: http://arxiv.org/abs/2604.07354v1
• Date: Sat, 28 Mar 2026 05:09:16 GMT
• Title: Contextual Earnings-22: A Speech Recognition Benchmark with Custom Vocabulary in the Wild
• Abstract summary: Speech-to-text accuracy has plateaued on academic benchmarks.<n>In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The accuracy frontier of speech-to-text systems has plateaued on academic benchmarks.1 In contrast, industrial benchmarks and adoption in high-stakes domains suggest otherwise. We hypothesize that the primary difference between the two is contextual conditioning: Academic benchmarks are dominated by frequently encountered general vocabulary that is relatively easy to recognize compared with rare and context-defined custom vocabulary that has disproportionate impact on the usability of speech transcripts. Despite progress on contextual speech-to-text, there is no standardized benchmark. We introduce Contextual Earnings-22, an open dataset built upon Earnings-22, with realistic custom vocabulary contexts to foster research and reveal latent progress. We set six strong baselines for two dominant approaches: keyword prompting and keyword boosting. Experiments show both reach comparable and significantly improved accuracy when scaled from proof-of-concept to large-scale systems.

Related papers

• How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs [61.601626186678146]
  We evaluate two context biasing methods based on Whisper against three speech large language models (LLMs)<n>The context biasing methods cut biased WER by up to 88% relative while leaving other words largely unaffected.<n> Speech LLMs excel on read speech but generalize less well to non-read speech, and prove sensitive to distractor count and prompt word order.
  arXiv  Detail & Related papers  (2026-08-06T08:45:33Z)
• Which Words Matter Most in Zero-Shot Prompts? [16.347012287506253]
  ZIP score is the first systematic method to quantify individual word importance in instructional prompts.<n>We show that task-specific word hierarchies exist where mathematical problems prioritize "step-by-step" while reasoning tasks favor "think"<n>We establish the first ground-truth benchmark for prompt interpretability through 20 validation prompts with predetermined key words.
  arXiv  Detail & Related papers  (2025-02-05T18:04:29Z)
• How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System? [7.252894835396412]
  Simultaneous speech-to-text translation (SimulST) translates source-language speech into target-language text concurrently with the speaker's speech, ensuring low latency for better user comprehension.<n>Despite its intended application to unbounded speech, most research has focused on human pre-segmented speech, simplifying the task and overlooking significant challenges.
  arXiv  Detail & Related papers  (2024-12-24T15:26:31Z)
• Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases [79.07111754406841]
  This work proposes using contrastive evaluation to measure the ability of direct S2TT systems to disambiguate utterances where prosody plays a crucial role. Our results clearly demonstrate the value of direct translation systems over cascade translation models.
  arXiv  Detail & Related papers  (2024-02-01T14:46:35Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• Incorporating Context into Subword Vocabularies [3.22352610570206]
  SaGe is a tokenizer that tailors subwords for downstream use by baking in the contextualized signal at the vocabulary creation phase. We show that SaGe does a better job than current widespread tokenizers in keeping token contexts cohesive.
  arXiv  Detail & Related papers  (2022-10-13T15:22:59Z)
• UCPhrase: Unsupervised Context-aware Quality Phrase Tagging [63.86606855524567]
  UCPhrase is a novel unsupervised context-aware quality phrase tagger. We induce high-quality phrase spans as silver labels from consistently co-occurring word sequences. We show that our design is superior to state-of-the-art pre-trained, unsupervised, and distantly supervised methods.
  arXiv  Detail & Related papers  (2021-05-28T19:44:24Z)
• Improving Machine Reading Comprehension with Contextualized Commonsense Knowledge [62.46091695615262]
  We aim to extract commonsense knowledge to improve machine reading comprehension. We propose to represent relations implicitly by situating structured knowledge in a context. We employ a teacher-student paradigm to inject multiple types of contextualized knowledge into a student machine reader.
  arXiv  Detail & Related papers  (2020-09-12T17:20:01Z)
• On Vocabulary Reliance in Scene Text Recognition [79.21737876442253]
  Methods perform well on images with words within vocabulary but generalize poorly to images with words outside vocabulary. We call this phenomenon "vocabulary reliance" We propose a simple yet effective mutual learning strategy to allow models of two families to learn collaboratively.
  arXiv  Detail & Related papers  (2020-05-08T11:16:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.