M3ST: Mix at Three Levels for Speech Translation

• URL: http://arxiv.org/abs/2212.03657v1
• Date: Wed, 7 Dec 2022 14:22:00 GMT
• Title: M3ST: Mix at Three Levels for Speech Translation
• Authors: Xuxin Cheng, Qianqian Dong, Fengpeng Yue, Tom Ko, Mingxuan Wang, Yuexian Zou
• Abstract summary: We propose Mix at three levels for Speech Translation (M3ST) method to increase the diversity of the augmented training corpus. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. Experiments on MuST-C speech translation benchmark and analysis show that M3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
• Score: 66.71994367650461
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.

Related papers

• CoT-ST: Enhancing LLM-based Speech Translation with Multimodal Chain-of-Thought [33.32415197728357]
  Speech Language Models (SLMs) have demonstrated impressive performance on speech translation tasks. We introduce a three-stage training framework designed to activate the chain-of-thought capabilities of SLMs. We propose CoT-ST, a speech translation model that utilizes multimodal CoT to decompose speech translation into sequential steps of speech recognition and translation.
  arXiv  Detail & Related papers  (2024-09-29T01:48:09Z)
• Improving speech translation by fusing speech and text [24.31233927318388]
  We harness the complementary strengths of speech and text, which are disparate modalities. We propose textbfFuse-textbfSpeech-textbfText (textbfFST), a cross-modal model which supports three distinct input modalities for translation.
  arXiv  Detail & Related papers  (2023-05-23T13:13:48Z)
• On the Pareto Front of Multilingual Neural Machine Translation [123.94355117635293]
  We study how the performance of a given direction changes with its sampling ratio in Neural Machine Translation (MNMT) We propose the Double Power Law to predict the unique performance trade-off front in MNMT. In our experiments, it achieves better performance than temperature searching and gradient manipulation methods with only 1/5 to 1/2 of the total training budget.
  arXiv  Detail & Related papers  (2023-04-06T16:49:19Z)
• Improving Simultaneous Machine Translation with Monolingual Data [94.1085601198393]
  Simultaneous machine translation (SiMT) is usually done via sequence-level knowledge distillation (Seq-KD) from a full-sentence neural machine translation (NMT) model. We propose to leverage monolingual data to improve SiMT, which trains a SiMT student on the combination of bilingual data and external monolingual data distilled by Seq-KD.
  arXiv  Detail & Related papers  (2022-12-02T14:13:53Z)
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation [61.564874831498145]
  TranSpeech is a speech-to-speech translation model with bilateral perturbation. We establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices. TranSpeech shows a significant improvement in inference latency, enabling speedup up to 21.4x than autoregressive technique.
  arXiv  Detail & Related papers  (2022-05-25T06:34:14Z)
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation [76.13334392868208]
  Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue.
  arXiv  Detail & Related papers  (2022-04-06T17:59:22Z)
• STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation [37.51435498386953]
  We propose the Speech-TExt Manifold Mixup (STEMM) method to calibrate such discrepancy. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy.
  arXiv  Detail & Related papers  (2022-03-20T01:49:53Z)
• Regularizing End-to-End Speech Translation with Triangular Decomposition Agreement [27.87144563354033]
  We propose a novel regularization method for model training to improve the agreement of dual-path decomposition within triplet data. Experiments on the MuST-C benchmark demonstrate that our proposed approach significantly outperforms state-of-the-art E2E-ST baselines.
  arXiv  Detail & Related papers  (2021-12-21T05:24:01Z)
• Consecutive Decoding for Speech-to-text Translation [51.155661276936044]
  COnSecutive Transcription and Translation (COSTT) is an integral approach for speech-to-text translation. The key idea is to generate source transcript and target translation text with a single decoder. Our method is verified on three mainstream datasets.
  arXiv  Detail & Related papers  (2020-09-21T10:10:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.