Whisper-UT: A Unified Translation Framework for Speech and Text

• URL: http://arxiv.org/abs/2509.16375v1
• Date: Fri, 19 Sep 2025 19:36:04 GMT
• Title: Whisper-UT: A Unified Translation Framework for Speech and Text
• Authors: Cihan Xiao, Matthew Wiesner, Debashish Chakraborty, Reno Kriz, Keith Cunningham, Kenton Murray, Kevin Duh, Luis Tavarez-Arce, Paul McNamee, Sanjeev Khudanpur,
• Abstract summary: Whisper-UT is a unified framework that leverages lightweight adapters to enable seamless adaptation across tasks.<n>We demonstrate our methods using the Whisper model, though in principle they are general and could be applied to similar multitask models.
• Score: 33.08182294010418
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Encoder-decoder models have achieved remarkable success in speech and text tasks, yet efficiently adapting these models to diverse uni/multi-modal scenarios remains an open challenge. In this paper, we propose Whisper-UT, a unified and efficient framework that leverages lightweight adapters to enable seamless adaptation across tasks, including a multi-modal machine translation (MMT) task that explicitly conditions translation on both speech and source language text inputs. By incorporating ASR hypotheses or ground-truth transcripts as prompts, this approach not only enables the system to process both modalities simultaneously but also enhances speech translation (ST) performance through a 2-stage decoding strategy. We demonstrate our methods using the Whisper model, though in principle they are general and could be applied to similar multitask models. We highlight the effectiveness of cross-modal and cross-task fine-tuning, which improves performance without requiring 3-way parallel data. Our results underscore the flexibility, efficiency, and general applicability of the proposed framework for multi-modal translation.

Related papers

• Unifying Speech Recognition, Synthesis and Conversion with Autoregressive Transformers [8.890811356340953]
  General-Purpose Audio (GPA) is a unified audio foundation model that integrates multiple core speech tasks within a single large language model (LLM) architecture.<n>GPA operates on a shared discrete audio token space and supports instruction-driven task induction, enabling a single autoregressive model to flexibly TTS, ASR, and VC.
  arXiv  Detail & Related papers  (2026-01-15T13:47:55Z)
• AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation [41.74261260212531]
  multimodal-to-speech task has gained increasing attention due to its wide range of applications, such as film production, dubbing, and virtual avatars.<n>Existing methods still suffer from limitations in speech intelligibility, audio-video synchronization, speech naturalness, and voice similarity to the reference speaker.<n>We propose AlignDiT, a multimodal Aligned Diffusion Transformer that generates accurate, synchronized, and natural-sounding speech from aligned multimodal inputs.
  arXiv  Detail & Related papers  (2025-04-29T10:56:24Z)
• Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture [14.056534007451763]
  Simultaneous speech translation (SimulST) produces translations incrementally while processing partial speech input.<n>Existing LLM-based SimulST approaches incur significant computational overhead due to repeated encoding of bidirectional speech encoder.<n>We introduce Efficient and Adaptive Simultaneous Speech Translation (EASiST) with fully unidirectional architecture.
  arXiv  Detail & Related papers  (2025-04-16T06:46:15Z)
• Zipper: A Multi-Tower Decoder Architecture for Fusing Modalities [6.9522425458326635]
  We propose a multi-tower decoder architecture that flexibly composes multimodal generative models from independently pre-trained unimodal decoders. We show the proposed architecture performs very competitively in scenarios with limited aligned text-speech data. In cross-modal tasks such as text-to-speech generation (TTS) where the output modality is speech, we show that using a pre-trained speech backbone results in superior performance to the baseline.
  arXiv  Detail & Related papers  (2024-05-29T00:23:55Z)
• WavLLM: Towards Robust and Adaptive Speech Large Language Model [93.0773293897888]
  We introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set.
  arXiv  Detail & Related papers  (2024-03-31T12:01:32Z)
• TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languages [92.86083489187403]
  Tri-Modal Translation (TMT) model translates between arbitrary modalities spanning speech, image, and text.<n>We tokenize speech and image data into discrete tokens, which provide a unified interface across modalities.<n>TMT outperforms single model counterparts consistently.
  arXiv  Detail & Related papers  (2024-02-25T07:46:57Z)
• Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing [72.56219471145232]
  We propose a ST/MT multi-tasking framework with hard parameter sharing. Our method reduces the speech-text modality gap via a pre-processing stage. We show that our framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU.
  arXiv  Detail & Related papers  (2023-09-27T17:48:14Z)
• VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation [91.39949385661379]
  VioLA is a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text. We first convert all the speech utterances to discrete tokens using an offline neural encoder. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks.
  arXiv  Detail & Related papers  (2023-05-25T14:39:47Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.