ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speech

• URL: http://arxiv.org/abs/2601.12289v1
• Date: Sun, 18 Jan 2026 07:05:40 GMT
• Title: ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speech
• Authors: Haowei Lou, Hye-young Paik, Wen Hu, Lina Yao,
• Abstract summary: ParaMETA is a framework for learning and controlling speaking styles directly from speech.<n>It learns disentangled, task-specific embeddings by projecting speech into dedicated subspaces for each type of style.<n>It supports both speech- and text-based prompting and allows users to modify one speaking style while preserving others.
• Score: 15.969757677847504
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Learning representative embeddings for different types of speaking styles, such as emotion, age, and gender, is critical for both recognition tasks (e.g., cognitive computing and human-computer interaction) and generative tasks (e.g., style-controllable speech generation). In this work, we introduce ParaMETA, a unified and flexible framework for learning and controlling speaking styles directly from speech. Unlike existing methods that rely on single-task models or cross-modal alignment, ParaMETA learns disentangled, task-specific embeddings by projecting speech into dedicated subspaces for each type of style. This design reduces inter-task interference, mitigates negative transfer, and allows a single model to handle multiple paralinguistic tasks such as emotion, gender, age, and language classification. Beyond recognition, ParaMETA enables fine-grained style control in Text-To-Speech (TTS) generative models. It supports both speech- and text-based prompting and allows users to modify one speaking styles while preserving others. Extensive experiments demonstrate that ParaMETA outperforms strong baselines in classification accuracy and generates more natural and expressive speech, while maintaining a lightweight and efficient model suitable for real-world applications.

Related papers

• F-Actor: Controllable Conversational Behaviour in Full-Duplex Models [70.48189107402145]
  We present first open, instruction-following full-stage conversational speech model that can be trained efficiently under typical academic resource constraints.<n>Our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage pretraining.<n>Both the model and training code will be released to enable reproducible research on controllable full-like controllable full-stage speech systems.
  arXiv  Detail & Related papers  (2026-01-16T14:25:57Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• VStyle: A Benchmark for Voice Style Adaptation with Spoken Instructions [66.93932684284695]
  Spoken language models (SLMs) have emerged as a unified paradigm for speech understanding and generation.<n>We introduce Voice Style Adaptation (VSA), a new task that examines whether SLMs can modify their speaking style.<n>We present VStyle, a benchmark covering four categories of speech generation: acoustic attributes, natural language instruction, role play, and implicit empathy.<n>We also introduce the Large Audio Language Model as a Judge (LALM as a Judge) framework, which progressively evaluates outputs along textual faithfulness, style adherence, and naturalness.
  arXiv  Detail & Related papers  (2025-09-09T14:28:58Z)
• Vision-Speech Models: Teaching Speech Models to Converse about Images [67.62394024470528]
  We introduce MoshiVis, augmenting a recent dialogue speech LLM, Moshi, with visual inputs through lightweight adaptation modules.<n>An additional dynamic gating mechanism enables the model to more easily switch between the visual inputs and unrelated conversation topics.<n>We evaluate the model on downstream visual understanding tasks with both audio and text prompts, and report qualitative samples of interactions with MoshiVis.
  arXiv  Detail & Related papers  (2025-03-19T18:40:45Z)
• LAST: Language Model Aware Speech Tokenization [24.185165710384997]
  We propose a novel approach to training a speech tokenizer by leveraging objectives from pre-trained textual LMs. Our aim is to transform features from a pre-trained speech model into a new feature space that enables better clustering for speech LMs.
  arXiv  Detail & Related papers  (2024-09-05T16:57:39Z)
• Learning Speech Representation From Contrastive Token-Acoustic Pretraining [57.08426714676043]
  We propose "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space. The proposed CTAP model is trained on 210k speech and phoneme pairs, achieving minimally-supervised TTS, VC, and ASR.
  arXiv  Detail & Related papers  (2023-09-01T12:35:43Z)
• ZS-MSTM: Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding [3.609538870261841]
  We propose a machine learning approach to synthesize gestures, driven by prosodic features and text, in the style of different speakers. Our model incorporates zero-shot multimodal style transfer using multimodal data from the PATS database.
  arXiv  Detail & Related papers  (2023-05-22T10:10:35Z)
• Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation [63.561944239071615]
  StyleSpeech is a new TTS model which synthesizes high-quality speech and adapts to new speakers. With SALN, our model effectively synthesizes speech in the style of the target speaker even from single speech audio. We extend it to Meta-StyleSpeech by introducing two discriminators trained with style prototypes, and performing episodic training.
  arXiv  Detail & Related papers  (2021-06-06T15:34:11Z)
• Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identity [21.61168067832304]
  We present an automatic gesture generation model that uses the multimodal context of speech text, audio, and speaker identity to reliably generate gestures. Experiments with the introduced metric and subjective human evaluation showed that the proposed gesture generation model is better than existing end-to-end generation models.
  arXiv  Detail & Related papers  (2020-09-04T11:42:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.