Listen to the Features: Voice Anonymization Driven by Content Embedding Matching over Signal Reconstruction

• URL: http://arxiv.org/abs/2607.09767v1
• Date: Tue, 07 Jul 2026 09:11:20 GMT
• Title: Listen to the Features: Voice Anonymization Driven by Content Embedding Matching over Signal Reconstruction
• Abstract summary: The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech.<n>It relies on content embeddings extracted from a frozen pretrained wav2vec2 encoder.<n>The training objective enforces that embeddings of the anonymized signal match those of the original one.
• Score: 0.6157382820537719
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paper presents a voice anonymization model focusing on preserving content rather than producing realistic speech. It relies on content embeddings extracted from a frozen pretrained wav2vec2 encoder. These embeddings are decoded into an anonymized signal using vector quantization and a HiFi-GAN vocoder, both trained on LibriTTS without any waveform reconstruction loss or speaker embedding mapping. The training objective enforces that embeddings of the anonymized signal match those of the original one. While training, an auxiliary speaker classification branch with a gradient reversal layer is used to discard speakerspecific information. Results show that this straightforward embedding-based approach achieves very low WER (2.53) with an anonymization performance (EER 13.39) ranking within first level for VPC. Notably, emotions are partially preserved (UAR 43.91), even without a supporting training objective, while the anonymized voice is audible without reconstruction loss.

Related papers

• SignDeepSC: A Semantic Signature-based Approach for Robust Semantic Communication [4.042562775811428]
  SignDeepSC is an architectural defense that achieves adversarial robustness without requiring adversarial example generation during training.<n>System is built on a perceiver-inspired semantic signature, a compact vector summary of the source features transmitted over a separate low-rate auxiliary channel.
  arXiv  Detail & Related papers  (2026-07-28T12:54:13Z)
• Making the Most of Limited Data: Score-Aware Training for Text-to-Music Generation [1.0323063834827415]
  State-of-the-art text-to-music generation systems rely on massive proprietary datasets and industrial-scale compute.<n>We propose textitscore-aware training, which treats audio-caption alignment score as a direct supervision signal throughout the pipeline.
  arXiv  Detail & Related papers  (2026-06-05T15:24:36Z)
• Privacy-Preserving Semantic Communications via Multi-Task Learning and Adversarial Perturbations [49.494000480610225]
  This paper presents a deep learning-based semantic communication framework.<n>It supports multiple receiver tasks while explicitly limiting semantic leakage to an eavesdropper.<n>A perturbation layer is successful in reducing semantic leakage even when the legitimate link is trained only for its own task.
  arXiv  Detail & Related papers  (2025-12-30T20:19:49Z)
• Transcription-Free Fine-Tuning of Speech Separation Models for Noisy and Reverberant Multi-Speaker Automatic Speech Recognition [18.50957174600796]
  Solution to automatic speech recognition (ASR) of overlapping speakers is to separate speech and then perform ASR on the separated signals. Currently, the separator produces artefacts which often degrade ASR performance. This paper proposes a transcription-free method for joint training using only audio signals.
  arXiv  Detail & Related papers  (2024-06-13T08:20:58Z)
• Fill in the Gap! Combining Self-supervised Representation Learning with Neural Audio Synthesis for Speech Inpainting [14.402357651227003]
  We investigate the use of a speech SSL model for speech inpainting, that is reconstructing a missing portion of a speech signal from its surrounding context. To that purpose, we combine an SSL encoder, namely HuBERT, with a neural vocoder, namely HiFiGAN, playing the role of a decoder.
  arXiv  Detail & Related papers  (2024-05-30T14:41:39Z)
• Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping [4.271091833712731]
  We propose a simple approach, named Lip2Vec that is based on learning a prior model. The proposed model compares favorably with fully-supervised learning methods on the LRS3 dataset achieving 26 WER. We believe that reprogramming the VSR as an ASR task narrows the performance gap between the two and paves the way for more flexible formulations of lip reading.
  arXiv  Detail & Related papers  (2023-08-11T12:59:02Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data [145.95460945321253]
  We introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes. The proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training.
  arXiv  Detail & Related papers  (2022-03-31T15:33:56Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• Word Order Does Not Matter For Speech Recognition [35.96275156733138]
  We train a word-level acoustic model which aggregates the distribution of all output frames. We then train a letter-based acoustic model using Connectionist Temporal Classification loss. Our system achieves 2.4%/5.3% on test-clean/test-other subsets of LibriSpeech.
  arXiv  Detail & Related papers  (2021-10-12T13:35:01Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention [66.77490220410249]
  We propose FragmentVC, in which the latent phonetic structure of the utterance from the source speaker is obtained from Wav2Vec 2.0. FragmentVC is able to extract fine-grained voice fragments from the target speaker utterance(s) and fuse them into the desired utterance. This approach is trained with reconstruction loss only without any disentanglement considerations between content and speaker information.
  arXiv  Detail & Related papers  (2020-10-27T09:21:03Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.