A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models

• URL: http://arxiv.org/abs/2607.00946v1
• Date: Wed, 01 Jul 2026 13:46:16 GMT
• Title: A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
• Abstract summary: We present the first comparative study of speech language model (SLM) and conditional flow-matching modules as activation steering sites for mixed emotion speech synthesis.<n>Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement.
• Score: 17.011586874604742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While prior work has explored emotion control in hybrid text-to-speech systems, the geometric properties of these modules, and their implications for steerability, remain poorly understood. We present the first comparative study of speech language model (SLM) and conditional flow-matching (CFM) modules as activation steering sites for mixed emotion speech synthesis. We first characterize emotion representations using linear probing and local intrinsic dimensionality (LID), and then evaluate single-site and joint steering for mixed-emotion synthesis. Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement. Joint steering increases emotion intensity but degrades proportional control and speech quality on in-distribution data. These findings provide practical guidance for multi-site activation steering in hybrid TTS systems and highlight the importance of representation geometry in controllable speech generation.

Related papers

• EII-SCL: Harnessing Emotional Inertia for Multimodal Emotion Recognition in Conversation [51.04910642896281]
  Multimodal emotion recognition in conversation (MERC) achieves accurate predictions by integrating multimodal and contextual information in dialogues.<n>Current MERC approaches overlook the impact of contextual emotional inertia in emotion shift, leading to sub-optimal performance.<n>We propose a novel Emotional Inertia-Informed Supervised Contrastive Learning module (EII-SCL) that informs the contrastive objective by constructing inertia-affected samples within temporal windows.
  arXiv  Detail & Related papers  (2026-07-19T18:09:54Z)
• Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS [61.29502937013759]
  We introduce a flow-matching based text-to-speech model trained with vocal effort and articulation pseudo-labels.<n>The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance.
  arXiv  Detail & Related papers  (2026-06-22T11:14:22Z)
• Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech [11.976178882036466]
  We analyze emotion-related variation in the semantic hidden states of large language models (LLMs) based text-to-speech systems.<n>Our analysis shows that emotional variation is distributed across multiple sparse latent features, while intervening on a small subset enables interpretable emotion control.<n>We introduce a feature-level intervention framework for bidirectional emotion induction and suppression without modifying backbone parameters.
  arXiv  Detail & Related papers  (2026-05-31T22:39:45Z)
• ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignment [52.31881685769569]
  We propose ImmersiveTTS, an environment-aware text-to-speech (TTS) model that generates natural speech seamlessly integrated within environmental contexts.<n>Our model builds on a multimodal diffusion transformer and fuses transcript-aligned speech latent with text-conditioned environmental context.<n> Experimental results show that ImmersiveTTS achieves higher naturalness, intelligibility, and audio fidelity than existing approaches.
  arXiv  Detail & Related papers  (2026-05-29T07:58:54Z)
• CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering [25.10244503397448]
  Emotional expression in human speech is nuanced and compositional, often involving multiple, conflicting, affective cues.<n>Most expressive text-to-speech systems enforce a single utterance-level emotion, collapsing affective diversity and suppressing mixed or text-emotion-misaligned expression.<n>This paper introduces a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-emotion synthesis and reliable text-emotion mismatch synthesis.
  arXiv  Detail & Related papers  (2026-02-03T11:45:00Z)
• Emotional Text-To-Speech Based on Mutual-Information-Guided Emotion-Timbre Disentanglement [37.959531845352274]
  Experimental results demonstrate that our method outperforms baseline TTS systems in generating natural and emotionally rich speech.<n>This work highlights the potential of disentangled and fine-grained representations in advancing the quality and flexibility of emotional TTS systems.
  arXiv  Detail & Related papers  (2025-10-02T07:03:50Z)
• Contrastive Decoupled Representation Learning and Regularization for Speech-Preserving Facial Expression Manipulation [58.189703277322224]
  Speech-preserving facial expression manipulation (SPFEM) aims to modify a talking head to display a specific reference emotion.<n>Emotion and content information existing in reference and source inputs can provide direct and accurate supervision signals for SPFEM models.<n>We propose to learn content and emotion priors as guidance augmented with contrastive learning to learn decoupled content and emotion representation.
  arXiv  Detail & Related papers  (2025-04-08T04:34:38Z)
• GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations [35.63053777817013]
  GatedxLSTM is a novel multimodal Emotion Recognition in Conversation (ERC) model.<n>It considers voice and transcripts of both the speaker and their conversational partner to identify the most influential sentences driving emotional shifts.<n>It achieves state-of-the-art (SOTA) performance among open-source methods in four-class emotion classification.
  arXiv  Detail & Related papers  (2025-03-26T18:46:18Z)
• UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts [64.02363948840333]
  UMETTS is a novel framework that leverages emotional cues from multiple modalities to generate highly expressive and emotionally resonant speech.<n>EP-Align employs contrastive learning to align emotional features across text, audio, and visual modalities, ensuring a coherent fusion of multimodal information.<n>EMI-TTS integrates the aligned emotional embeddings with state-of-the-art TTS models to synthesize speech that accurately reflects the intended emotions.
  arXiv  Detail & Related papers  (2024-04-29T03:19:39Z)
• Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion [81.1492897350032]
  Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. We propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion.
  arXiv  Detail & Related papers  (2023-12-29T08:06:45Z)
• ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models [83.07390037152963]
  ZET-Speech is a zero-shot adaptive emotion-controllable TTS model. It allows users to synthesize any speaker's emotional speech using only a short, neutral speech segment and the target emotion label. Experimental results demonstrate that ZET-Speech successfully synthesizes natural and emotional speech with the desired emotion for both seen and unseen speakers.
  arXiv  Detail & Related papers  (2023-05-23T08:52:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.