Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis

• URL: http://arxiv.org/abs/2503.04814v1
• Date: Tue, 04 Mar 2025 15:28:10 GMT
• Title: Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis
• Authors: Yiming Wang, Yi Yang, Jiahong Yuan,
• Abstract summary: This study investigates the normalization process within transformer models, especially wav2vec 2.0.<n>We found that fine-tuning wav2vec 2.0 effectively achieves phonetic normalization by selectively suppressing task-irrelevant information.<n>These findings provide new insights into how phonetic normalization can be flexibly achieved in speech models and how it is realized in human speech perception.
• Score: 32.14451400240806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Phonetic normalization plays a crucial role in speech recognition and analysis, ensuring the comparability of features derived from raw audio data. However, in the current paradigm of fine-tuning pre-trained large transformer models, phonetic normalization is not deemed a necessary step; instead, it is implicitly executed within the models. This study investigates the normalization process within transformer models, especially wav2vec 2.0. Through a comprehensive analysis of embeddings from models fine-tuned for various tasks, our results demonstrate that fine-tuning wav2vec 2.0 effectively achieves phonetic normalization by selectively suppressing task-irrelevant information. We found that models fine-tuned for multiple tasks retain information for both tasks without compromising performance, and that suppressing task-irrelevant information is not necessary for effective classification. These findings provide new insights into how phonetic normalization can be flexibly achieved in speech models and how it is realized in human speech perception.

Related papers

• Automatic classification of stop realisation with wav2vec2.0 [0.24466725954625887]
  We show that wav2vec2.0 models can be trained to automatically classify stop burst presence with high accuracy in both English and Japanese.<n>Results demonstrate the potential of pre-trained speech models as tools for the automatic annotation and processing of speech corpus data.
  arXiv  Detail & Related papers  (2025-05-29T17:25:35Z)
• Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model [0.0]
  OpenAI's Whisper Automated Speech Recognition model excels in generalizing across diverse datasets and domains. We propose a method to enhance transcription accuracy without explicit fine-tuning or altering model parameters.
  arXiv  Detail & Related papers  (2024-10-24T01:58:11Z)
• Unveiling and Mitigating Bias in Audio Visual Segmentation [9.427676046134374]
  Community researchers have developed a range of advanced audio-visual segmentation models to improve the quality of sounding objects' masks. While masks created by these models may initially appear plausible, they occasionally exhibit anomalies with incorrect grounding logic. We attribute this to real-world inherent preferences and distributions as a simpler signal for learning than the complex audio-visual grounding.
  arXiv  Detail & Related papers  (2024-07-23T16:55:04Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation [72.7915031238824]
  Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks. They often suffer from common issues such as semantic misalignment and poor temporal consistency. We propose Make-an-Audio 2, a latent diffusion-based T2A method that builds on the success of Make-an-Audio.
  arXiv  Detail & Related papers  (2023-05-29T10:41:28Z)
• AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations [88.30635799280923]
  We introduce AV-data2vec which builds audio-visual representations based on predicting contextualized representations. Results on LRS3 show that AV-data2vec consistently outperforms existing methods with the same amount of data and model size.
  arXiv  Detail & Related papers  (2023-02-10T02:55:52Z)
• Fully Automated End-to-End Fake Audio Detection [57.78459588263812]
  This paper proposes a fully automated end-toend fake audio detection method. We first use wav2vec pre-trained model to obtain a high-level representation of the speech. For the network structure, we use a modified version of the differentiable architecture search (DARTS) named light-DARTS.
  arXiv  Detail & Related papers  (2022-08-20T06:46:55Z)
• Analyzing Robustness of End-to-End Neural Models for Automatic Speech Recognition [11.489161072526677]
  We investigate robustness properties of pre-trained neural models for automatic speech recognition. In this work, we perform a robustness analysis of the pre-trained neural models wav2vec2, HuBERT and DistilHuBERT on the LibriSpeech and TIMIT datasets.
  arXiv  Detail & Related papers  (2022-08-17T20:00:54Z)
• WavThruVec: Latent speech representation as intermediate features for neural speech synthesis [1.1470070927586016]
  WavThruVec is a two-stage architecture that resolves the bottleneck by using high-dimensional Wav2Vec 2.0 embeddings as intermediate speech representation. We show that the proposed model not only matches the quality of state-of-the-art neural models, but also presents useful properties enabling tasks like voice conversion or zero-shot synthesis.
  arXiv  Detail & Related papers  (2022-03-31T10:21:08Z)
• Voice2Series: Reprogramming Acoustic Models for Time Series Classification [65.94154001167608]
  Voice2Series is a novel end-to-end approach that reprograms acoustic models for time series classification. We show that V2S either outperforms or is tied with state-of-the-art methods on 20 tasks, and improves their average accuracy by 1.84%.
  arXiv  Detail & Related papers  (2021-06-17T07:59:15Z)
• Switching Variational Auto-Encoders for Noise-Agnostic Audio-visual Speech Enhancement [26.596930749375474]
  We introduce the use of a latent sequential variable with Markovian dependencies to switch between different VAE architectures through time. We derive the corresponding variational expectation-maximization algorithm to estimate the parameters of the model and enhance the speech signal.
  arXiv  Detail & Related papers  (2021-02-08T11:45:02Z)
• Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations [49.55361944105796]
  We present a novel approach to any-to-one (A2O) voice conversion (VC) in a sequence-to-sequence framework. A2O VC aims to convert any speaker, including those unseen during training, to a fixed target speaker.
  arXiv  Detail & Related papers  (2020-10-23T08:34:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.