DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition

• URL: http://arxiv.org/abs/2501.13497v1
• Date: Thu, 23 Jan 2025 09:25:31 GMT
• Title: DQ-Data2vec: Decoupling Quantization for Multilingual Speech Recognition
• Authors: Qijie Shao, Linhao Dong, Kun Wei, Sining Sun, Lei Xie,
• Abstract summary: Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction. We propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. DQ-Data2vec achieves a relative reduction of 9.51% in phoneme error rate (PER) and 11.58% in word error rate (WER) compared to data2vec and UniData2vec.
• Score: 26.129088764394588
• License:
• Abstract: Data2vec is a self-supervised learning (SSL) approach that employs a teacher-student architecture for contextual representation learning via masked prediction, demonstrating remarkable performance in monolingual ASR. Previous studies have revealed that data2vec's shallow layers capture speaker and language information, middle layers encode phoneme and word features, while deep layers are responsible for reconstruction. Language and phoneme features are crucial for multilingual ASR. However, data2vec's masked representation generation relies on multi-layer averaging, inevitably coupling these features. To address this limitation, we propose a decoupling quantization based data2vec (DQ-Data2vec) for multilingual ASR, which includes a data2vec backbone and two improved online K-means quantizers. Our core idea is using the K-means quantizer with specified cluster numbers to decouple language and phoneme information for masked prediction. Specifically, in the language quantization, considering that the number of languages is significantly different from other irrelevant features (e.g., speakers), we assign the cluster number to match the number of languages, explicitly decoupling shallow layers' language-related information from irrelevant features. This strategy is also applied to decoupling middle layers' phoneme and word features. In a self-supervised scenario, experiments on the CommonVoice dataset demonstrate that DQ-Data2vec achieves a relative reduction of 9.51% in phoneme error rate (PER) and 11.58% in word error rate (WER) compared to data2vec and UniData2vec. Moreover, in a weakly-supervised scenario incorporating language labels and high-resource language text labels, the relative reduction is 18.09% and 1.55%, respectively.

Related papers

• Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching [65.74653592668743]
  Finetuning self-supervised multilingual representations reduces absolute word error rates by up to 20%. In circumstances with limited training data finetuning self-supervised representations is a better performing and viable solution.
  arXiv  Detail & Related papers  (2023-11-25T17:05:21Z)
• Transfer-Free Data-Efficient Multilingual Slot Labeling [82.02076369811402]
  Slot labeling is a core component of task-oriented dialogue (ToD) systems. To mitigate the inherent data scarcity issue, current research on multilingual ToD assumes that sufficient English-language annotated data are always available. We propose a two-stage slot labeling approach (termed TWOSL) which transforms standard multilingual sentence encoders into effective slot labelers.
  arXiv  Detail & Related papers  (2023-05-22T22:47:32Z)
• Mitigating Data Imbalance and Representation Degeneration in Multilingual Machine Translation [103.90963418039473]
  Bi-ACL is a framework that uses only target-side monolingual data and a bilingual dictionary to improve the performance of the MNMT model. We show that Bi-ACL is more effective both in long-tail languages and in high-resource languages.
  arXiv  Detail & Related papers  (2023-05-22T07:31:08Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setup [1.2031796234206138]
  We propose a new Self-Supervised Learning (SSL) algorithm called data2vec-aqc. Our goal is to improve SSL for speech in domains where both unlabeled and labeled data are limited.
  arXiv  Detail & Related papers  (2022-11-02T16:29:59Z)
• Pretraining Approaches for Spoken Language Recognition: TalTech Submission to the OLR 2021 Challenge [0.0]
  The paper is based on our submission to the Oriental Language Recognition 2021 Challenge. For the constrained track, we first trained a Conformer-based encoder-decoder model for multilingual automatic speech recognition. For the unconstrained task, we relied on both externally available pretrained models as well as external data.
  arXiv  Detail & Related papers  (2022-05-14T15:17:08Z)
• A Hierarchical Model for Spoken Language Recognition [29.948719321162883]
  Spoken language recognition ( SLR) refers to the automatic process used to determine the language present in a speech sample. We propose a novel hierarchical approach were two PLDA models are trained, one to generate scores for clusters of highly related languages and a second one to generate scores conditional to each cluster. We show that this hierarchical approach consistently outperforms the non-hierarchical one for detection of highly related languages.
  arXiv  Detail & Related papers  (2022-01-04T22:10:36Z)
• Low-Resource Spoken Language Identification Using Self-Attentive Pooling and Deep 1D Time-Channel Separable Convolutions [0.0]
  We show that a convolutional neural network with a Self-Attentive Pooling layer shows promising results in low-resource setting for the language identification task. We also substantiate the hypothesis that whenever the dataset is diverse enough so that the other classification factors, like gender, age etc. are well-averaged, the confusion matrix for LID system bears the language similarity measure.
  arXiv  Detail & Related papers  (2021-05-31T18:35:27Z)
• Efficient Weight factorization for Multilingual Speech Recognition [67.00151881207792]
  End-to-end multilingual speech recognition involves using a single model training on a compositional speech corpus including many languages. Due to the fact that each language in the training data has different characteristics, the shared network may struggle to optimize for all various languages simultaneously. We propose a novel multilingual architecture that targets the core operation in neural networks: linear transformation functions.
  arXiv  Detail & Related papers  (2021-05-07T00:12:02Z)
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders [111.65322283420805]
  We present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bi-directional EncodeR) AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model.
  arXiv  Detail & Related papers  (2020-10-15T18:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.