WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling

• URL: http://arxiv.org/abs/2605.06407v1
• Date: Thu, 07 May 2026 15:17:24 GMT
• Title: WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
• Abstract summary: WavCube is a compact continuous latent derived from an SSL speech encoder.<n>It simultaneously supports speech understanding, reconstruction, and generation.<n>Trials show it delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence.
• Score: 35.33131758542107
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrating speech understanding and generation is a pivotal step toward building unified speech models. However, the different representations required for these two tasks currently pose significant compatibility challenges. Typically, semantics-oriented features are learned from self-supervised learning (SSL), and acoustic-oriented features from reconstruction. Such fragmented representations hinder the realization of truly unified speech systems. We present WavCube, a compact continuous latent derived from an SSL speech encoder that simultaneously supports speech understanding, reconstruction, and generation. WavCube employs a two-stage training scheme. Stage 1 trains a semantic bottleneck to filter off-manifold redundancy that makes raw SSL features intractable for diffusion. Stage 2 injects fine-grained acoustic details via end-to-end reconstruction, while a semantic anchoring loss ensures the representation remains grounded within its original semantic manifold. Comprehensive experiments show that WavCube closely approaches WavLM performance on SUPERB despite an 8x dimensional compression, attains reconstruction quality on par with existing acoustic representations, delivers state-of-the-art zero-shot TTS performance with markedly faster training convergence, and excels in speech enhancement, separation, and voice conversion tasks on the SUPERB-SG benchmark. Systematic ablations reveal that WavCube's two-stage recipe resolves two intrinsic flaws of SSL features for generative modeling, paving the way for future unified speech systems. Codes and checkpoints are available at https://github.com/yanghaha0908/WavCube.

Related papers

• OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation [73.57253966578186]
  We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations.<n>Experiments show that both objectives consistently improve the learnability of the latent spaces.
  arXiv  Detail & Related papers  (2026-07-26T21:51:28Z)
• HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec [0.0]
  HybridCodec employs separate semantic and acoustic branches while distilling SSL representations into the semantic stream.<n>It shows superior semantic robustness (RVQ-1) on in-domain test set and competitive reconstruction (RVQ-all)<n>We demonstrate its specialization in out-of-domain and zero-shot cross-lingual settings, achieving a 3x speedup over existing dual-stream models.
  arXiv  Detail & Related papers  (2026-06-04T21:57:18Z)
• LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation [17.04920367653778]
  Existing unified tokenizers jointly encode both in high-dimensional continuous latents.<n>We propose LoSATok, a low-dimensional audio tokenizer for cross-domain audio understanding and generation.
  arXiv  Detail & Related papers  (2026-05-27T01:56:45Z)
• DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding [58.29124051111574]
  We introduce DIFFA-2, a practical diffusion-based LALM for general audio understanding.<n>DIFFA-2 upgrades the speech encoder, employs dual semantic and acoustic adapters, and is trained with a four-stage curriculum.<n>Experiments on MMSU, MMAU, and MMAR show that DIFFA-2 consistently improves over DIFFA.
  arXiv  Detail & Related papers  (2026-01-30T16:44:23Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis [57.5830191022097]
  A Text-to-Vec module generates Wav2Vec2 embeddings from text, which jointly condition speech and face generation.<n>We adopt a two-stage training: pretraining on Wav2Vec2 embeddings and finetuning on TTS outputs.<n>Experiments show that conditioning on TTS-predicted latent features outperforms cascaded pipelines.
  arXiv  Detail & Related papers  (2025-11-07T17:07:56Z)
• Fill in the Gap! Combining Self-supervised Representation Learning with Neural Audio Synthesis for Speech Inpainting [14.402357651227003]
  We investigate the use of a speech SSL model for speech inpainting, that is reconstructing a missing portion of a speech signal from its surrounding context. To that purpose, we combine an SSL encoder, namely HuBERT, with a neural vocoder, namely HiFiGAN, playing the role of a decoder.
  arXiv  Detail & Related papers  (2024-05-30T14:41:39Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• Minimally-Supervised Speech Synthesis with Conditional Diffusion Model and Language Model: A Comparative Study of Semantic Coding [57.42429912884543]
  We propose Diff-LM-Speech, Tetra-Diff-Speech and Tri-Diff-Speech to solve high dimensionality and waveform distortion problems. We also introduce a prompt encoder structure based on a variational autoencoder and a prosody bottleneck to improve prompt representation ability. Experimental results show that our proposed methods outperform baseline methods.
  arXiv  Detail & Related papers  (2023-07-28T11:20:23Z)
• Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR [35.710735895190844]
  We propose a self-supervised framework named Wav2code to implement a feature-level SE with reduced distortions for noise-robust ASR. During finetuning, we propose a Transformer-based code predictor to accurately predict clean codes by modeling the global dependency of input noisy representations. Experiments on both synthetic and real noisy datasets demonstrate that Wav2code can solve the speech distortion and improve ASR performance under various noisy conditions.
  arXiv  Detail & Related papers  (2023-04-11T04:46:12Z)
• TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation [61.564874831498145]
  TranSpeech is a speech-to-speech translation model with bilateral perturbation. We establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices. TranSpeech shows a significant improvement in inference latency, enabling speedup up to 21.4x than autoregressive technique.
  arXiv  Detail & Related papers  (2022-05-25T06:34:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.