Residual Tokens Enhance Masked Autoencoders for Speech Modeling

• URL: http://arxiv.org/abs/2601.19399v1
• Date: Tue, 27 Jan 2026 09:30:31 GMT
• Title: Residual Tokens Enhance Masked Autoencoders for Speech Modeling
• Authors: Samir Sadok, Stéphane Lathuilière, Xavier Alameda-Pineda,
• Abstract summary: We introduce RT-MAE, a novel masked autoencoder framework that augments the supervised attributes-based modeling with unsupervised residual trainable tokens.<n> Experiments show that RT-MAE improves reconstruction quality, preserving content and speaker similarity while enhancing expressivity.
• Score: 31.12667739896418
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent speech modeling relies on explicit attributes such as pitch, content, and speaker identity, but these alone cannot capture the full richness of natural speech. We introduce RT-MAE, a novel masked autoencoder framework that augments the supervised attributes-based modeling with unsupervised residual trainable tokens, designed to encode the information not explained by explicit labeled factors (e.g., timbre variations, noise, emotion etc). Experiments show that RT-MAE improves reconstruction quality, preserving content and speaker similarity while enhancing expressivity. We further demonstrate its applicability to speech enhancement, removing noise at inference while maintaining controllability and naturalness.

Related papers

• Frontend Token Enhancement for Token-Based Speech Recognition [50.35062963870211]
  Discretized representations of speech signals are efficient alternatives to continuous features for speech recognition applications.<n>In this work, we introduce a system that estimates clean speech tokens from noisy speech and evaluate it on an ASR backend using semantic tokens.<n>We consider four types of enhancement models based on their input/token domains: wave-to-wave, token-to-output, continuous SSL features-to-token, and wave-to-token.
  arXiv  Detail & Related papers  (2026-02-04T05:02:15Z)
• Autoregressive Speech Enhancement via Acoustic Tokens [12.77742493025067]
  We study the performance of acoustic tokens for speech enhancement and introduce a novel transducer-based autoregressive architecture.<n>Experiments on VoiceBank and Libri1 datasets show that acoustic tokens outperform semantic tokens in terms of preserving speaker identity.
  arXiv  Detail & Related papers  (2025-07-17T06:32:22Z)
• A Variational Framework for Improving Naturalness in Generative Spoken Language Models [52.673912922590866]
  We propose an end-to-end variational approach that automatically learns to encode continuous speech attributes to enhance semantic tokens.<n>Our approach eliminates the need for manual extraction and selection of paralinguistic features.<n>It produces preferred speech continuations according to human raters.
  arXiv  Detail & Related papers  (2025-06-17T17:58:17Z)
• Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility [15.463932957443973]
  Speech restoration aims at restoring full-band speech with high quality and intelligibility, considering a diverse set of distortions. MaskSR is a recently proposed generative model for this task. We show that, with the same MaskSR model capacity and inference time, the proposed model, MaskSR2, significantly reduces the word error rate, a typical metric for intelligibility.
  arXiv  Detail & Related papers  (2024-09-14T08:09:55Z)
• On the Semantic Latent Space of Diffusion-Based Text-to-Speech Models [15.068637971987224]
  We explore the latent space of frozen TTS models, which is composed of the latent bottleneck activations of the DDM's denoiser. We identify that this space contains rich semantic information, and outline several novel methods for finding semantic directions within it, both supervised and unsupervised. We demonstrate how these enable off-the-shelf audio editing, without any further training, architectural changes or data requirements.
  arXiv  Detail & Related papers  (2024-02-19T16:22:21Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models [57.71199494492223]
  We propose a Unified Speech Enhancement and Editing (uSee) model with conditional diffusion models to handle various tasks at the same time in a generative manner. Our experiments show that our proposed uSee model can achieve superior performance in both speech denoising and dereverberation compared to other related generative speech enhancement models.
  arXiv  Detail & Related papers  (2023-10-02T04:36:39Z)
• EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis [49.04496602282718]
  We introduce Expresso, a high-quality expressive speech dataset for textless speech synthesis. This dataset includes both read speech and improvised dialogues rendered in 26 spontaneous expressive styles. We evaluate resynthesis quality with automatic metrics for different self-supervised discrete encoders.
  arXiv  Detail & Related papers  (2023-08-10T17:41:19Z)
• Content-Context Factorized Representations for Automated Speech Recognition [12.618527387900079]
  We introduce an unsupervised, encoder-agnostic method for factoring speech-encoder representations into explicit content-encoding representations and spurious context-encoding representations. We demonstrate improved performance on standard ASR benchmarks, as well as improved performance in both real-world and artificially noisy ASR scenarios.
  arXiv  Detail & Related papers  (2022-05-19T21:34:40Z)
• Fine-grained Noise Control for Multispeaker Speech Synthesis [3.449700218265025]
  A text-to-speech (TTS) model typically factorizes speech attributes such as content, speaker and prosody into disentangled representations. Recent works aim to additionally model the acoustic conditions explicitly, in order to disentangle the primary speech factors.
  arXiv  Detail & Related papers  (2022-04-11T13:13:55Z)
• High Fidelity Speech Regeneration with Application to Speech Enhancement [96.34618212590301]
  We propose a wav-to-wav generative model for speech that can generate 24khz speech in a real-time manner. Inspired by voice conversion methods, we train to augment the speech characteristics while preserving the identity of the source.
  arXiv  Detail & Related papers  (2021-01-31T10:54:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.