Multilingual Source Tracing of Speech Deepfakes: A First Benchmark

• URL: http://arxiv.org/abs/2508.04143v1
• Date: Wed, 06 Aug 2025 07:11:36 GMT
• Title: Multilingual Source Tracing of Speech Deepfakes: A First Benchmark
• Authors: Xi Xuan, Yang Xiao, Rohan Kumar Das, Tomi Kinnunen,
• Abstract summary: This paper introduces the first benchmark for multilingual speech deepfake source tracing.<n>We comparatively investigate DSP- and SSL-based modeling, examine how SSL representations fine-tuned on different languages impact cross-lingual generalization performance.<n>Our findings offer the first comprehensive insights into the challenges of identifying speech generation models when training and inference languages differ.
• Score: 19.578741954970738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in generative AI has made it increasingly easy to create natural-sounding deepfake speech from just a few seconds of audio. While these tools support helpful applications, they also raise serious concerns by making it possible to generate convincing fake speech in many languages. Current research has largely focused on detecting fake speech, but little attention has been given to tracing the source models used to generate it. This paper introduces the first benchmark for multilingual speech deepfake source tracing, covering both mono- and cross-lingual scenarios. We comparatively investigate DSP- and SSL-based modeling; examine how SSL representations fine-tuned on different languages impact cross-lingual generalization performance; and evaluate generalization to unseen languages and speakers. Our findings offer the first comprehensive insights into the challenges of identifying speech generation models when training and inference languages differ. The dataset, protocol and code are available at https://github.com/xuanxixi/Multilingual-Source-Tracing.

Related papers

• Tell me Habibi, is it Real or Fake? [15.344187517040508]
  Code-switching, especially between Arabic and English, is common in the Arab world and is widely used in digital communication.<n>We introduce textbfArEnAV, the first large-scale Arabic-English audio-visual deepfake dataset featuring intra-utterance code-switching, dialectal variation, and monolingual Arabic content.<n>Our dataset is generated using a novel pipeline integrating four Text-To-Speech and two lip-sync models, enabling comprehensive analysis of multilingual multimodal deepfake detection.
  arXiv  Detail & Related papers  (2025-05-28T16:54:36Z)
• Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation [48.769137497536]
  We propose the unit language to overcome the two modeling challenges.<n>The unit language can be considered a text-like representation format.<n>We implement multi-task learning to utilize the unit language in guiding the speech modeling process.
  arXiv  Detail & Related papers  (2025-05-21T10:05:25Z)
• A two-stage transliteration approach to improve performance of a multilingual ASR [1.9511556030544333]
  This paper presents an approach to build a language-agnostic end-to-end model trained on a grapheme set. We performed experiments with an end-to-end multilingual speech recognition system for two Indic languages.
  arXiv  Detail & Related papers  (2024-10-09T05:30:33Z)
• Cross-Lingual Transfer Learning for Speech Translation [7.802021866251242]
  This paper examines how to expand the speech translation capability of speech foundation models with restricted data.<n>Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model.<n>Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space.
  arXiv  Detail & Related papers  (2024-07-01T09:51:48Z)
• AudioPaLM: A Large Language Model That Can Speak and Listen [79.44757696533709]
  We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models. It can process and generate text and speech with applications including speech recognition and speech-to-speech translation.
  arXiv  Detail & Related papers  (2023-06-22T14:37:54Z)
• Learning Cross-lingual Visual Speech Representations [108.68531445641769]
  Cross-lingual self-supervised visual representation learning has been a growing research topic in the last few years. We use the recently-proposed Raw Audio-Visual Speechs (RAVEn) framework to pre-train an audio-visual model with unlabelled data. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance.
  arXiv  Detail & Related papers  (2023-03-14T17:05:08Z)
• Speech-to-Speech Translation For A Real-world Unwritten Language [62.414304258701804]
  We study speech-to-speech translation (S2ST) that translates speech from one language into another language. We present an end-to-end solution from training data collection, modeling choices to benchmark dataset release.
  arXiv  Detail & Related papers  (2022-11-11T20:21:38Z)
• LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers [71.76680102779765]
  Automatic speech recognition (ASR) and speech translation (ST) can both use neural transducers as the model structure. We propose LAMASSU, a streaming language-agnostic multilingual speech recognition and translation model using neural transducers.
  arXiv  Detail & Related papers  (2022-11-05T04:03:55Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Speaker Independent and Multilingual/Mixlingual Speech-Driven Talking Head Generation Using Phonetic Posteriorgrams [58.617181880383605]
  In this work, we propose a novel approach using phonetic posteriorgrams. Our method doesn't need hand-crafted features and is more robust to noise compared to recent approaches. Our model is the first to support multilingual/mixlingual speech as input with convincing results.
  arXiv  Detail & Related papers  (2020-06-20T16:32:43Z)
• Cross-lingual Multispeaker Text-to-Speech under Limited-Data Scenario [10.779568857641928]
  This paper presents an extension on Tacotron2 to achieve bilingual multispeaker speech synthesis. We achieve cross-lingual synthesis, including code-switching cases, between English and Mandarin for monolingual speakers.
  arXiv  Detail & Related papers  (2020-05-21T03:03:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.