Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation

• URL: http://arxiv.org/abs/2508.11189v1
• Date: Fri, 15 Aug 2025 03:46:46 GMT
• Title: Novel Parasitic Dual-Scale Modeling for Efficient and Accurate Multilingual Speech Translation
• Authors: Chenyang Le, Yinfeng Xia, Huiyan Li, Manhong Wang, Yutao Sun, Xingyang Ma, Yanmin Qian,
• Abstract summary: We propose an innovative Parasitic Dual-Scale Approach, which combines an enhanced speculative sampling method with model compression and knowledge distillation techniques.<n>We build on the Whisper Medium model, and integrate our novel KVSPN module, achieving state-of-the-art (SOTA) performance across six popular languages with improved inference efficiency.
• Score: 33.87915567709033
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in speech-to-text translation have led to the development of multilingual models capable of handling multiple language pairs simultaneously. However, these unified models often suffer from large parameter sizes, making it challenging to balance inference efficiency and performance, particularly in local deployment scenarios. We propose an innovative Parasitic Dual-Scale Approach, which combines an enhanced speculative sampling method with model compression and knowledge distillation techniques. Building on the Whisper Medium model, we enhance it for multilingual speech translation into whisperM2M, and integrate our novel KVSPN module, achieving state-of-the-art (SOTA) performance across six popular languages with improved inference efficiency. KVSPN enables a 40\% speedup with no BLEU score degradation. Combined with distillation methods, it represents a 2.6$\times$ speedup over the original Whisper Medium with superior performance.

Related papers

• Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study [13.409987421121405]
  GemmaX2-28 is a 9B model achieving top-tier multilingual translation performance across 28 languages.<n>GemmaX2-28 consistently outperforms the state-of-the-art (SOTA) models such as TowerInstruct and XALMA.
  arXiv  Detail & Related papers  (2025-02-04T16:57:03Z)
• Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation [70.22782550540714]
  Quantization-aware Scale LeArning method based on multimodal Warmup, termed QSLAW. We introduce a Quantization-aware Scale LeArning method based on multimodal Warmup, termed QSLAW.
  arXiv  Detail & Related papers  (2024-08-07T12:42:09Z)
• Efficient Compression of Multitask Multilingual Speech Models [0.0]
  DistilWhisper is able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2.
  arXiv  Detail & Related papers  (2024-05-02T03:11:59Z)
• On the Analysis of Cross-Lingual Prompt Tuning for Decoder-based Multilingual Model [49.81429697921861]
  We study the interaction between parameter-efficient fine-tuning (PEFT) and cross-lingual tasks in multilingual autoregressive models. We show that prompt tuning is more effective in enhancing the performance of low-resource languages than fine-tuning.
  arXiv  Detail & Related papers  (2023-11-14T00:43:33Z)
• Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts [14.999359332108767]
  We propose DistilWhisper to bridge the performance gap in ASR for under-represented languages. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters.
  arXiv  Detail & Related papers  (2023-11-02T08:37:30Z)
• Exploiting Multilingualism in Low-resource Neural Machine Translation via Adversarial Learning [3.2258463207097017]
  Generative Adversarial Networks (GAN) offer a promising approach for Neural Machine Translation (NMT) In GAN, similar to bilingual models, multilingual NMT only considers one reference translation for each sentence during model training. This article proposes Denoising Adversarial Auto-encoder-based Sentence Interpolation (DAASI) approach to perform sentence computation.
  arXiv  Detail & Related papers  (2023-03-31T12:34:14Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models [81.84866217721361]
  DiffusionBERT is a new generative masked language model based on discrete diffusion models. We propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text.
  arXiv  Detail & Related papers  (2022-11-28T03:25:49Z)
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models [82.22294901727933]
  A minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability. Our approach demonstrates strong zero-shot performance in a many-to-many multilingual model.
  arXiv  Detail & Related papers  (2020-10-24T08:15:08Z)
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation [81.7786241489002]
  Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics. We propose random online backtranslation to enforce the translation of unseen training language pairs.
  arXiv  Detail & Related papers  (2020-04-24T17:21:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.