MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond

• URL: http://arxiv.org/abs/2412.11538v2
• Date: Fri, 20 Dec 2024 09:12:47 GMT
• Title: MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond
• Abstract summary: The MERaLiON-SpeechEncoder is a foundation model designed to support a wide range of downstream speech applications.<n>It was developed as part of Singapore's National Multimodal Large Language Model Programme.<n>The model currently supports mainly English, including the variety spoken in Singapore.
• Score: 33.46428078179056
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This technical report describes the MERaLiON-SpeechEncoder, a foundation model designed to support a wide range of downstream speech applications. Developed as part of Singapore's National Multimodal Large Language Model Programme, the MERaLiON-SpeechEncoder is tailored to address the speech processing needs in Singapore and the surrounding Southeast Asian region. The model currently supports mainly English, including the variety spoken in Singapore. We are actively expanding our datasets to gradually cover other languages in subsequent releases. The MERaLiON-SpeechEncoder was pre-trained from scratch on 200,000 hours of unlabelled speech data using a self-supervised learning approach based on masked language modelling. We describe our training procedure and hyperparameter tuning experiments in detail below. Our evaluation demonstrates improvements to spontaneous and Singapore speech benchmarks for speech recognition, while remaining competitive to other state-of-the-art speech encoders across ten other speech tasks. We commit to releasing our model, supporting broader research endeavours, both in Singapore and beyond.

Related papers

• SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia [20.33019395334099]
  We introduce SEA-SpeechBench, the first large-scale benchmark that evaluates speech understanding in 11 Southeast Asian languages.<n>Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding.<n>We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models.
  arXiv  Detail & Related papers  (2026-09-09T03:39:18Z)
• PRiSM: Benchmarking Phone Realization in Speech Models [70.82595415252682]
  Phone recognition (PR) serves as the atomic interface for language-agnostic modeling for cross-lingual speech processing and phonetic analysis.<n>We introduce PRiSM, the first open-source benchmark designed to expose blind spots in phonetic perception.
  arXiv  Detail & Related papers  (2026-01-20T15:00:36Z)
• Speech Language Models for Under-Represented Languages: Insights from Wolof [9.14632796153174]
  We present our journey in training a speech language model for Wolof, an underrepresented language spoken in West Africa.<n>We first emphasize the importance of collecting large-scale, spontaneous, high-quality unsupervised speech data.<n>We show that continued pretraining HuBERT on this dataset outperforms both the base model and African-centric models on ASR.
  arXiv  Detail & Related papers  (2025-09-18T19:01:48Z)
• SENSE models: an open source solution for multilingual and multimodal semantic-based tasks [11.951746684966205]
  SENSE (Shared Embedding for N-lingual Speech and tExt) is an open-source solution inspired by the SAMU-XLSR framework.<n>We describe how the original SAMU-XLSR method has been updated by selecting a stronger teacher text model and a better initial speech encoder.<n>We report experimental results on multilingual and multimodal semantic tasks, where our SENSE model achieves highly competitive performance.
  arXiv  Detail & Related papers  (2025-09-15T16:18:51Z)
• Advancing Singlish Understanding: Bridging the Gap with Datasets and Multimodal Models [38.608158064184366]
  We standardize and annotate the largest spoken Singlish corpus, introducing the Multitask National Speech Corpus (MNSC) These datasets support diverse tasks, including Automatic Speech Recognition (ASR), Spoken Question Answering (SQA), Spoken Dialogue Summarization (SDS) and Paralinguistic Question Answering (PQA) We propose SingAudioLLM, a multi-task multimodal model leveraging multimodal large language models to handle these tasks concurrently.
  arXiv  Detail & Related papers  (2025-01-02T03:28:52Z)
• MERaLiON-AudioLLM: Technical Report [37.90189271403642]
  We introduce MERaLiON-AudioLLM, the first speech-text model tailored for Singapore's multicultural landscape.<n> MERaLiON-AudioLLM integrates advanced speech and text processing to address the diverse linguistic nuances of local accents and dialects.
  arXiv  Detail & Related papers  (2024-12-13T03:15:05Z)
• Cross-Lingual Transfer Learning for Speech Translation [7.802021866251242]
  This paper examines how to expand the speech translation capability of speech foundation models with restricted data. Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model. Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space.
  arXiv  Detail & Related papers  (2024-07-01T09:51:48Z)
• CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving [61.73180469072787]
  We focus on the problem of spoken translation (ST) of code-switched speech in Indian languages to English text. We present a new end-to-end model architecture COSTA that scaffolds on pretrained automatic speech recognition (ASR) and machine translation (MT) modules. COSTA significantly outperforms many competitive cascaded and end-to-end multimodal baselines by up to 3.5 BLEU points.
  arXiv  Detail & Related papers  (2024-06-16T16:10:51Z)
• TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation [97.54885207518946]
  We introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion. We propose two separated encoders to preserve the speaker's voice characteristics and isochrony from the source speech during the translation process. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.
  arXiv  Detail & Related papers  (2024-05-28T04:11:37Z)
• OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification [44.94458898538114]
  We propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC) It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID) Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 24% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference.
  arXiv  Detail & Related papers  (2024-02-20T02:04:38Z)
• PolyVoice: Language Models for Speech to Speech Translation [50.31000706309143]
  PolyVoice is a language model-based framework for speech-to-speech translation (S2ST) We use discretized speech units, which are generated in a fully unsupervised way. For the speech synthesis part, we adopt the existing VALL-E X approach and build a unit-based audio language model.
  arXiv  Detail & Related papers  (2023-06-05T15:53:15Z)
• Textless Speech-to-Speech Translation With Limited Parallel Data [51.3588490789084]
  PFB is a framework for training textless S2ST models that require just dozens of hours of parallel speech data. We train and evaluate our models for English-to-German, German-to-English and Marathi-to-English translation on three different domains.
  arXiv  Detail & Related papers  (2023-05-24T17:59:05Z)
• The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translation [13.352795145385645]
  Speech translation (ST) is a good means of pretraining speech models for end-to-end spoken language understanding. We show that our models reach higher performance over baselines on monolingual and multilingual intent classification. We also create new benchmark datasets for speech summarization and low-resource/zero-shot transfer from English to French or Spanish.
  arXiv  Detail & Related papers  (2023-05-16T17:53:03Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Mandarin-English Code-switching Speech Recognition with Self-supervised Speech Representation Models [55.82292352607321]
  Code-switching (CS) is common in daily conversations where more than one language is used within a sentence. This paper uses the recently successful self-supervised learning (SSL) methods to leverage many unlabeled speech data without CS.
  arXiv  Detail & Related papers  (2021-10-07T14:43:35Z)
• KARI: KAnari/QCRI's End-to-End systems for the INTERSPEECH 2021 Indian Languages Code-Switching Challenge [7.711092265101041]
  We present the Kanari/QCRI system and the modeling strategies used to participate in the Interspeech 2021 Code-switching (CS) challenge for low-resource Indian languages. The subtask involved developing a speech recognition system for two CS datasets: Hindi-English and Bengali-English. To tackle the CS challenges, we use transfer learning for incorporating the publicly available monolingual Hindi, Bengali, and English speech data.
  arXiv  Detail & Related papers  (2021-06-10T16:12:51Z)
• Multilingual Bottleneck Features for Improving ASR Performance of Code-Switched Speech in Under-Resourced Languages [12.139300459657974]
  We explore the benefits of using multilingual bottleneck features (mBNF) in acoustic modelling for the automatic speech recognition of code-switched (CS) speech in African languages. We train a mBNF extractor using nine Southern Bantu languages that form part of the freely available multilingual NCHLT corpus. Our results show that the inclusion of the mBNF features leads to clear performance improvements over a baseline trained without the mBNFs for code-switched English-isiZulu, English-isiXa, English-Sesotho and English-Setswana speech.
  arXiv  Detail & Related papers  (2020-10-31T18:51:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.