POWSM: A Phonetic Open Whisper-Style Speech Foundation Model

• URL: http://arxiv.org/abs/2510.24992v1
• Date: Tue, 28 Oct 2025 21:43:45 GMT
• Title: POWSM: A Phonetic Open Whisper-Style Speech Foundation Model
• Authors: Chin-Jou Li, Kalvin Chang, Shikhar Bharadwaj, Eunjung Yeo, Kwanghee Choi, Jian Zhu, David Mortensen, Shinji Watanabe,
• Abstract summary: POWSM is the first unified framework capable of jointly performing multiple phone-related tasks.<n>Our training data, code and models are released to foster open science.
• Score: 50.73202227472358
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in spoken language processing have led to substantial progress in phonetic tasks such as automatic speech recognition (ASR), phone recognition (PR), grapheme-to-phoneme conversion (G2P), and phoneme-to-grapheme conversion (P2G). Despite their conceptual similarity, these tasks have largely been studied in isolation, each relying on task-specific architectures and datasets. In this paper, we introduce POWSM (Phonetic Open Whisper-style Speech Model), the first unified framework capable of jointly performing multiple phone-related tasks. POWSM enables seamless conversion between audio, text (graphemes), and phones, opening up new possibilities for universal and low-resource speech processing. Our model outperforms or matches specialized PR models of similar size (Wav2Vec2Phoneme and ZIPA) while jointly supporting G2P, P2G, and ASR. Our training data, code and models are released to foster open science.

Related papers

• DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations [62.00227663434538]
  DRVOICE-7B establishes new state-of-the-art (SOTA) on OpenAudioBench and Big Bench Audio benchmarks.<n>This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling.
  arXiv  Detail & Related papers  (2025-06-11T02:57:22Z)
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models [74.80386066714229]
  We present an improved streaming speech synthesis model, CosyVoice 2.<n>Specifically, we introduce finite-scalar quantization to improve codebook utilization of speech tokens.<n>We develop a chunk-aware causal flow matching model to support various synthesis scenarios.
  arXiv  Detail & Related papers  (2024-12-13T12:59:39Z)
• Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer [39.31849739010572]
  We introduce textbfGenerative textbfPre-trained textbfSpeech textbfTransformer (GPST) GPST is a hierarchical transformer designed for efficient speech language modeling.
  arXiv  Detail & Related papers  (2024-06-03T04:16:30Z)
• Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks [61.3055230762097]
  We propose a decoder-only language model, VoxtLM, that can perform four tasks: speech recognition, speech synthesis, text generation, and speech continuation. VoxtLM integrates text vocabulary with discrete speech tokens from self-supervised speech features and uses special tokens to enable multitask learning.
  arXiv  Detail & Related papers  (2023-09-14T03:13:18Z)
• Improving grapheme-to-phoneme conversion by learning pronunciations from speech recordings [12.669655363646257]
  The Grapheme-to-Phoneme (G2P) task aims to convert orthographic input into a discrete phonetic representation. We propose a method to improve the G2P conversion task by learning pronunciation examples from audio recordings.
  arXiv  Detail & Related papers  (2023-07-31T13:25:38Z)
• AudioPaLM: A Large Language Model That Can Speak and Listen [79.44757696533709]
  We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models. It can process and generate text and speech with applications including speech recognition and speech-to-speech translation.
  arXiv  Detail & Related papers  (2023-06-22T14:37:54Z)
• Text-Free Prosody-Aware Generative Spoken Language Modeling [46.19240899818964]
  We present a prosody-aware generative spoken language model (pGSLM) It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to waveforms. Experimental results show that the pGSLM can utilize prosody to improve both prosody and content modeling, and also generate natural, meaningful, and coherent speech given a spoken prompt.
  arXiv  Detail & Related papers  (2021-09-07T18:03:21Z)
• RECOApy: Data recording, pre-processing and phonetic transcription for end-to-end speech-based applications [4.619541348328938]
  RECOApy streamlines the steps of data recording and pre-processing required in end-to-end speech-based applications. The tool implements an easy-to-use interface for prompted speech recording, spectrogram and waveform analysis, utterance-level normalisation and silence trimming. The grapheme-to-phoneme (G2P) converters are deep neural network (DNN) based architectures trained on lexicons extracted from the Wiktionary online collaborative resource.
  arXiv  Detail & Related papers  (2020-09-11T15:26:55Z)
• Universal Phone Recognition with a Multilingual Allophone System [135.2254086165086]
  We propose a joint model of language-independent phone and language-dependent phoneme distributions. In multilingual ASR experiments over 11 languages, we find that this model improves testing performance by 2% phoneme error rate absolute. Our recognizer achieves phone accuracy improvements of more than 17%, moving a step closer to speech recognition for all languages in the world.
  arXiv  Detail & Related papers  (2020-02-26T21:28:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.