Deepfake Audio Detection Using Self-supervised Fusion Representations

• URL: http://arxiv.org/abs/2605.03420v1
• Date: Tue, 05 May 2026 06:51:41 GMT
• Title: Deepfake Audio Detection Using Self-supervised Fusion Representations
• Abstract summary: This paper describes a submission to the Environment-Aware Speech and Sound Deepfake Detection Challenge (ESDD2) 2026.<n>A dual-branch deepfake detection framework is proposed to jointly model speech and environmental contextual representations from input audio.<n>The proposed system achieves an F1-score of 70.20% and an environmental EER of 16.54%, outperforming the baseline system.
• Score: 5.9998333708296245
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper describes a submission to the Environment-Aware Speech and Sound Deepfake Detection Challenge (ESDD2) 2026, which addresses component-level deepfake detection using the CompSpoofV2 dataset, where speech and environmental sounds may be independently manipulated. To address this challenge, a dual-branch deepfake detection framework is proposed to jointly model speech and environmental contextual representations from input audio. Two pretrained models, XLS-R for speech and BEATs for environmental sound, are used to extract complementary contextual representations. A Matching Head is introduced to model representation differences through statistical normalization and representation interaction, enabling estimation of the original class. In parallel, multi-head cross-attention enables effective information exchange between speech and environmental components. The refined representations are processed with residual connections and layer normalization, and passed to an AASIST classifier to predict speech-based and environment-based spoofing probabilities. The model outputs original, speech, and environment predictions. On the test set, the proposed system achieves an F1-score of 70.20% and an environmental EER of 16.54%, outperforming the baseline system.

Related papers

• Teffic-Audio: Tell Fact from Fiction [15.841023625898053]
  This report presents Teffic-Audio, a general speech deepfake detection system for comprehensive evaluation environment.<n>Teffic-Audio adopts a straightforward detector architecture consisting of a Conformer-based speech encoder, multi-head attentive statistics pooling, and a binary classifier.<n>Teffic-Audio achieves a pooled EER of 1.454% on the 14 test sets of Speech-DF-Arena, outperforming all currently public systems on the leaderboard.
  arXiv  Detail & Related papers  (2026-07-30T15:21:38Z)
• ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment [52.31881685769569]
  We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts.<n>Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with text-conditioned environmental context.<n> Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches.
  arXiv  Detail & Related papers  (2026-05-29T07:58:54Z)
• SpatialEmb: Extract and Encode Spatial Information for 1-Stage Multi-channel Multi-speaker ASR on Arbitrary Microphone Arrays [45.93777164579776]
  We propose a lightweight embedding module named SpatialEmb, which extracts and encodes spatial information directly for the ASR model.<n>We conduct comprehensive experiments on AliMeeting, a real meeting corpus, to determine the optimal model design for SpatialEmb.<n>Our best model trained with 105 hours achieves 17.04% and 20.32% character error rates (CER) on the Eval and Test sets.
  arXiv  Detail & Related papers  (2026-01-25T23:21:49Z)
• Explainable Transformer-CNN Fusion for Noise-Robust Speech Emotion Recognition [2.0391237204597363]
  Speech Emotion Recognition systems often degrade in performance when exposed to unpredictable acoustic interference.<n>We propose a Hybrid Transformer-CNN framework that unifies the contextual modeling of Wav2Vec 2.0 with the spectral stability of 1D-Convolutional Neural Networks.
  arXiv  Detail & Related papers  (2025-12-20T10:05:58Z)
• V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow [57.51550409392103]
  We introduce V2SFlow, a novel Video-to-Speech (V2S) framework designed to generate natural and intelligible speech directly from silent talking face videos.<n>To address these challenges, we decompose the speech signal into manageable subspaces, each representing distinct speech attributes, and predict them directly from the visual input.<n>To generate coherent and realistic speech from these predicted attributes, we employ a rectified flow matching decoder built on a Transformer architecture.
  arXiv  Detail & Related papers  (2024-11-29T05:55:20Z)
• Exposing Synthetic Speech: Model Attribution and Detection of AI-generated Speech via Audio Fingerprints [11.703509488782345]
  We introduce a training-free, yet effective approach for detecting AI-generated speech.<n>We tackle three key tasks: (1) single-model attribution in an open-world setting, (2) multi-model attribution in a closed-world setting, and (3) detection of synthetic versus real speech.
  arXiv  Detail & Related papers  (2024-11-21T10:55:49Z)
• Describe Where You Are: Improving Noise-Robustness for Speech Emotion Recognition with Text Description of the Environment [28.491885755907575]
  Speech emotion recognition (SER) systems often struggle in real-world environments, where ambient noise severely degrades their performance.<n>This paper explores a novel approach that exploits prior knowledge of testing environments to maximize SER performance under noisy conditions.
  arXiv  Detail & Related papers  (2024-07-25T02:30:40Z)
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension [95.8442896569132]
  We introduce AIR-Bench, the first benchmark to evaluate the ability of Large Audio-Language Models (LALMs) to understand various types of audio signals and interact with humans in the textual format. Results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation.
  arXiv  Detail & Related papers  (2024-02-12T15:41:22Z)
• ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition [100.30565531246165]
  Speech recognition systems require dataset-specific tuning. This tuning requirement can lead to systems failing to generalise to other datasets and domains. We introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition system.
  arXiv  Detail & Related papers  (2022-10-24T15:58:48Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Visualizing Classifier Adjacency Relations: A Case Study in Speaker Verification and Voice Anti-Spoofing [72.4445825335561]
  We propose a simple method to derive 2D representation from detection scores produced by an arbitrary set of binary classifiers. Based upon rank correlations, our method facilitates a visual comparison of classifiers with arbitrary scores. While the approach is fully versatile and can be applied to any detection task, we demonstrate the method using scores produced by automatic speaker verification and voice anti-spoofing systems.
  arXiv  Detail & Related papers  (2021-06-11T13:03:33Z)
• Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding [76.89426311082927]
  Existing models are trained on clean data, which causes a textitgap between clean data training and real-world inference. We propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Experiments on the widely-used dataset, Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment.
  arXiv  Detail & Related papers  (2021-04-13T17:54:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.