Optimizing Byte-level Representation for End-to-end ASR

• URL: http://arxiv.org/abs/2406.09676v2
• Date: Wed, 4 Sep 2024 20:04:05 GMT
• Title: Optimizing Byte-level Representation for End-to-end ASR
• Abstract summary: We propose a novel approach to optimize a byte-level representation for end-to-end automatic speech recognition (ASR) Byte-level representation is often used by large scale ASR systems when the character set of the supported languages is large. In an English/Mandarin dictation task, we show that a bilingual ASR model built with this approach can outperform by 5% relative in error rate.
• Score: 4.247168417243709
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a novel approach to optimizing a byte-level representation for end-to-end automatic speech recognition (ASR). Byte-level representation is often used by large scale multilingual ASR systems when the character set of the supported languages is large. The compactness and universality of byte-level representation allow the ASR models to use smaller output vocabularies and therefore, provide more flexibility. UTF-8 is a commonly used byte-level representation for multilingual ASR, but it is not designed to optimize machine learning tasks directly. By using auto-encoder and vector quantization, we show that we can optimize a byte-level representation for ASR and achieve better accuracy. Our proposed framework can incorporate information from different modalities, and provides an error correction mechanism. In an English/Mandarin dictation task, we show that a bilingual ASR model built with this approach can outperform UTF-8 representation by 5% relative in error rate.

Related papers

• Linguistically Informed Tokenization Improves ASR for Underresourced Languages [0.0]
  We fine-tune a wav2vec2 ASR model on Yan-nhangu, a dormant Indigenous Australian language.<n>A linguistically informed phonemic tokenization system substantially improves WER and CER.<n>Hand-correcting the output of an ASR model is much faster than hand-transcribing audio from scratch, demonstrating that ASR can work for underresourced languages.
  arXiv  Detail & Related papers  (2025-10-07T20:54:35Z)
• NL in the Middle: Code Translation with LLMs and Intermediate Representations [66.41928783565795]
  Large language models (LLMs) produce buggy code translations.<n>We consider whether code translation using LLMs can benefit from intermediate representations via natural language (NL) and abstract syntax trees (ASTs)
  arXiv  Detail & Related papers  (2025-07-11T14:29:21Z)
• Improving Language and Modality Transfer in Translation by Character-level Modeling [14.145120349133007]
  Current translation systems, despite being highly multilingual, cover only 5% of the world's languages.<n>We propose a character-based approach to improve adaptability to new languages and modalities.
  arXiv  Detail & Related papers  (2025-05-30T13:16:08Z)
• Configurable Multilingual ASR with Speech Summary Representations [5.989153210779794]
  Half of the world's population is multilingual, making multilingual ASR (MASR) essential. deploying multiple monolingual models is challenging when the ground-truth language is unknown in advance. We present a novel csvMASR architecture designed to enhance configurability.
  arXiv  Detail & Related papers  (2024-10-06T13:39:15Z)
• Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking [68.77659513993507]
  We present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy. Our results show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively, and word error rates which are 3.3% and 2.0% lower on these benchmarks.
  arXiv  Detail & Related papers  (2024-09-27T03:31:32Z)
• ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets [106.7760874400261]
  This paper presents ML-SUPERB2.0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models. We find performance improvements over the setup of ML-SUPERB, but performance depends on the downstream model design. Also, we find large performance differences between languages and datasets, suggesting the need for more targeted approaches.
  arXiv  Detail & Related papers  (2024-06-12T21:01:26Z)
• Unified model for code-switching speech recognition and language identification based on a concatenated tokenizer [17.700515986659063]
  Code-Switching (CS) multilingual Automatic Speech Recognition (ASR) models can transcribe speech containing two or more alternating languages during a conversation. This paper proposes a new method for creating code-switching ASR datasets from purely monolingual data sources. A novel Concatenated Tokenizer enables ASR models to generate language ID for each emitted text token while reusing existing monolingual tokenizers.
  arXiv  Detail & Related papers  (2023-06-14T21:24:11Z)
• Efficient Spoken Language Recognition via Multilabel Classification [53.662747523872305]
  We show that our models obtain competitive results while being orders of magnitude smaller and faster than current state-of-the-art methods. Our multilabel strategy is more robust to unseen non-target languages compared to multiclass classification.
  arXiv  Detail & Related papers  (2023-06-02T23:04:19Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition [50.93943755401025]
  We propose a new parameter-efficient learning framework based on neural model reprogramming for cross-lingual speech recognition. We design different auxiliary neural architectures focusing on learnable pre-trained feature enhancement. Our methods outperform existing ASR tuning architectures and their extension with self-supervised losses.
  arXiv  Detail & Related papers  (2023-01-19T02:37:56Z)
• Learning ASR pathways: A sparse multilingual ASR model [31.147484652643282]
  We present ASR pathways, a sparse multilingual ASR model that activates language-specific sub-networks ("pathways") With the overlapping sub-networks, the shared parameters can also enable knowledge transfer for lower-resource languages via joint multilingual training. Our proposed ASR pathways outperform both dense models and a language-agnostically pruned model, and provide better performance on low-resource languages.
  arXiv  Detail & Related papers  (2022-09-13T05:14:08Z)
• Bilingual End-to-End ASR with Byte-Level Subwords [4.268218327369146]
  We study different representations including character-level, byte-level, byte pair encoding (BPE), and byte-level byte pair encoding (BBPE) We focus on developing a single end-to-end model to support utterance-based bilingual ASR, where speakers do not alternate between two languages in a single utterance but may change languages across utterances. We find that BBPE with penalty schemes can improve utterance-based bilingual ASR performance by 2% to 5% relative even with smaller number of outputs and fewer parameters.
  arXiv  Detail & Related papers  (2022-05-01T15:01:01Z)
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation [63.16500026845157]
  We introduce speech-to-text translation as an auxiliary task to incorporate additional knowledge of the target language. We show that training ST with human translations is not necessary. Even with pseudo-labels from low-resource MT (200K examples), ST-enhanced transfer brings up to 8.9% WER reduction to direct transfer.
  arXiv  Detail & Related papers  (2020-06-09T19:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.