OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer

• URL: http://arxiv.org/abs/2401.16658v3
• Date: Tue, 27 Aug 2024 02:15:49 GMT
• Title: OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
• Authors: Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee-weon Jung, Shinji Watanabe,
• Abstract summary: The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper using public data and open-source toolkits. We present a series of E-Branchformer-based models named OWSM v3.1, ranging from 100M to 1B parameters. OWSM v3.1 outperforms its predecessor, OWSM v3, in most evaluation benchmarks, while showing an improved inference speed of up to 25%.
• Score: 67.75820725013372
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies have highlighted the importance of fully open foundation models. The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper using public data and open-source toolkits. However, previous versions of OWSM (v1 to v3) are still based on standard Transformer, which might lead to inferior performance compared to state-of-the-art speech encoder architectures. This work aims to improve the performance and efficiency of OWSM without additional data. We present a series of E-Branchformer-based models named OWSM v3.1, ranging from 100M to 1B parameters. OWSM v3.1 outperforms its predecessor, OWSM v3, in most evaluation benchmarks, while showing an improved inference speed of up to 25%. We further reveal the emergent ability of OWSM v3.1 in zero-shot contextual biasing speech recognition. We also provide a model trained on a subset of data with low license restrictions. We will publicly release the code, pre-trained models, and training logs.

Related papers

• MooER: LLM-based Speech Recognition and Translation Models from Moore Threads [13.02816167879662]
  MooER is a large-scale automatic speech recognition (ASR) / automatic speech translation (AST) model of Moore Threads. A 5000h pseudo labeled dataset containing open source and self collected speech data is used for training. Experiments conducted on Covost2 Zh2en testset suggest that our model outperforms other open source Speech LLMs.
  arXiv  Detail & Related papers  (2024-08-09T14:43:56Z)
• On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models [57.97940182536942]
  The Open Whisper-style Speech Model (OWSM) series was introduced to achieve full transparency in building advanced speech-to-text (S2T) foundation models. OWSM models are trained on 25 public speech datasets, which are heterogeneous in multiple ways. We introduce OWSM v3.2, which improves on prior models by investigating and addressing the impacts of this data heterogeneity.
  arXiv  Detail & Related papers  (2024-06-13T16:22:37Z)
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance [68.18779562801762]
  multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance. Our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
  arXiv  Detail & Related papers  (2024-04-04T17:58:02Z)
• OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification [44.94458898538114]
  We propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC) It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID) Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 24% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference.
  arXiv  Detail & Related papers  (2024-02-20T02:04:38Z)
• Exploring the limits of decoder-only models trained on public speech recognition corpora [36.446905777292066]
  Decoder-Only Transformer for ASR (DOTA) model comprehensively outperforms the encoder-decoder open source replication of Whisper (OWSM) on nearly all English ASR benchmarks and outperforms Whisper large-v3 on 7 out of 15 test sets.
  arXiv  Detail & Related papers  (2024-01-31T23:29:42Z)
• Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data [75.7383558074758]
  This work presents an Open Whisper-style Speech Model (OWSM) OWSM reproduces Whisper-style training using an open-source toolkit and publicly available data. We will publicly release all scripts used for data preparation, training, inference, and scoring as well as pre-trained models and training logs to promote open science.
  arXiv  Detail & Related papers  (2023-09-25T05:01:34Z)
• The False Promise of Imitating Proprietary LLMs [158.65692029352584]
  An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model. This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes. We then evaluate the models using crowd raters and canonical NLP benchmarks.
  arXiv  Detail & Related papers  (2023-05-25T05:00:12Z)
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing [117.41016786835452]
  This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model. vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. We propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics.
  arXiv  Detail & Related papers  (2021-11-18T06:48:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.