CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

• URL: http://arxiv.org/abs/2305.14635v2
• Date: Thu, 25 May 2023 08:55:41 GMT
• Title: CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation
• Authors: Yan Zhou, Qingkai Fang, Yang Feng
• Abstract summary: End-to-end speech translation (ST) is a cross-modal task. Existing methods often try to transfer knowledge from machine translation (MT) We propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality gap.
• Score: 15.139447549817483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text. Code is publicly available at https://github.com/ictnlp/CMOT.

Related papers

• TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages [96.8603701943286]
  Tri-Modal Translation (TMT) model translates between arbitrary modalities spanning speech, image, and text. We tokenize speech and image data into discrete tokens, which provide a unified interface across modalities. TMT outperforms single model counterparts consistently.
  arXiv  Detail & Related papers  (2024-02-25T07:46:57Z)
• Pushing the Limits of Zero-shot End-to-End Speech Translation [15.725310520335785]
  Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems. We introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without any paired ST data. Our experiments show that we can effectively close the modality gap without ST data, while our results on MuST-C and CoVoST demonstrate our method's superiority.
  arXiv  Detail & Related papers  (2024-02-16T03:06:37Z)
• Soft Alignment of Modality Space for End-to-end Speech Translation [49.29045524083467]
  End-to-end Speech Translation aims to convert speech into target text within a unified model. The inherent differences between speech and text modalities often impede effective cross-modal and cross-lingual transfer. We introduce Soft Alignment (S-Align), using adversarial training to align the representation spaces of both modalities.
  arXiv  Detail & Related papers  (2023-12-18T06:08:51Z)
• Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing [72.56219471145232]
  We propose a ST/MT multi-tasking framework with hard parameter sharing. Our method reduces the speech-text modality gap via a pre-processing stage. We show that our framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU.
  arXiv  Detail & Related papers  (2023-09-27T17:48:14Z)
• Understanding and Bridging the Modality Gap for Speech Translation [11.13240570688547]
  Multi-task learning is one of the effective ways to share knowledge between machine translation (MT) and end-to-end speech translation (ST) However, due to the differences between speech and text, there is always a gap between ST and MT. In this paper, we first aim to understand this modality gap from the target-side representation differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias.
  arXiv  Detail & Related papers  (2023-05-15T15:09:18Z)
• M3ST: Mix at Three Levels for Speech Translation [66.71994367650461]
  We propose Mix at three levels for Speech Translation (M3ST) method to increase the diversity of the augmented training corpus. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. Experiments on MuST-C speech translation benchmark and analysis show that M3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
  arXiv  Detail & Related papers  (2022-12-07T14:22:00Z)
• Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation [71.35243644890537]
  End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions. Existing zero-shot methods fail to align the two modalities of speech and text into a shared semantic space. We propose a novel Discrete Cross-Modal Alignment (DCMA) method that employs a shared discrete vocabulary space to accommodate and match both modalities of speech and text.
  arXiv  Detail & Related papers  (2022-10-18T03:06:47Z)
• Generating Synthetic Speech from SpokenVocab for Speech Translation [18.525896864903416]
  Training end-to-end speech translation systems requires sufficiently large-scale data. One practical solution is to convert machine translation data (MT) to ST data via text-to-speech (TTS) systems. We propose a simple, scalable and effective data augmentation technique, i.e., SpokenVocab, to convert MT data to ST data on-the-fly.
  arXiv  Detail & Related papers  (2022-10-15T03:07:44Z)
• STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation [37.51435498386953]
  We propose the Speech-TExt Manifold Mixup (STEMM) method to calibrate such discrepancy. Experiments on MuST-C speech translation benchmark and further analysis show that our method effectively alleviates the cross-modal representation discrepancy.
  arXiv  Detail & Related papers  (2022-03-20T01:49:53Z)
• Consecutive Decoding for Speech-to-text Translation [51.155661276936044]
  COnSecutive Transcription and Translation (COSTT) is an integral approach for speech-to-text translation. The key idea is to generate source transcript and target translation text with a single decoder. Our method is verified on three mainstream datasets.
  arXiv  Detail & Related papers  (2020-09-21T10:10:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.