Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech

• URL: http://arxiv.org/abs/2606.01479v1
• Date: Sun, 31 May 2026 22:39:45 GMT
• Title: Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech
• Abstract summary: We analyze emotion-related variation in the semantic hidden states of large language models (LLMs) based text-to-speech systems.<n>Our analysis shows that emotional variation is distributed across multiple sparse latent features, while intervening on a small subset enables interpretable emotion control.<n>We introduce a feature-level intervention framework for bidirectional emotion induction and suppression without modifying backbone parameters.
• Score: 11.976178882036466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Integrating large language models (LLMs) into text-to-speech (TTS) systems has improved speech expressiveness, yet interpretable emotional control remains challenging. Existing approaches primarily rely on external conditioning or global activation steering, offering limited insight into the internal representations underlying emotional control. In this work, we analyze emotion-related variation in the semantic hidden states of LLM-based TTS models using sparse autoencoders (SAEs) to identify sparse latent features. Our analysis shows that emotional variation is distributed across multiple sparse latent features, while intervening on a small subset enables interpretable emotion control. Building on this observation, we introduce a feature-level intervention framework for bidirectional emotion induction and suppression without modifying backbone parameters. We further show that distinct latent features are associated with specific acoustic attributes (e.g., pitch), suggesting that emotional expression arises from coordinated latent contributions rather than a single global shift. Empirically, steering these sparse latent features achieves comparable or superior emotion induction and suppression performance relative to global steering and existing TTS baselines.

Related papers

• EII-SCL: Harnessing Emotional Inertia for Multimodal Emotion Recognition in Conversation [51.04910642896281]
  Multimodal emotion recognition in conversation (MERC) achieves accurate predictions by integrating multimodal and contextual information in dialogues.<n>Current MERC approaches overlook the impact of contextual emotional inertia in emotion shift, leading to sub-optimal performance.<n>We propose a novel Emotional Inertia-Informed Supervised Contrastive Learning module (EII-SCL) that informs the contrastive objective by constructing inertia-affected samples within temporal windows.
  arXiv  Detail & Related papers  (2026-07-19T18:09:54Z)
• A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models [17.011586874604742]
  We present the first comparative study of speech language model (SLM) and conditional flow-matching modules as activation steering sites for mixed emotion speech synthesis.<n>Our results show that SLM offers a clean, low-dimensional emotion-specific subspace with strong speaker--emotion disentanglement, while CFM exhibitspoor cross-speaker generalization due to speaker--emotion entanglement.
  arXiv  Detail & Related papers  (2026-07-01T13:46:16Z)
• From Syntax to Emotion: A Mechanistic Analysis of Emotion Inference in LLMs [1.4692712025142225]
  We investigate the internal mechanisms of emotion recognition in large language models (LLMs)<n>By analyzing sparse feature activations across layers, we identify a consistent three-phase information flow.<n>We show that emotion representations comprise both shared features across emotions and emotion-specific features.
  arXiv  Detail & Related papers  (2026-04-28T17:03:16Z)
• Linguistic Signatures for Enhanced Emotion Detection [0.802904964931021]
  This study examines whether linguistic features can serve as reliable interpretable signals for emotion recognition in text.<n>We extract emotion-specific linguistic signatures from 13 English datasets and evaluate how incorporating these features into transformer models impacts performance.<n>Our RoBERTa-based models enriched with high level linguistic features achieve consistent performance gains of up to +2.4 macro F1 on the GoEmotions benchmark.
  arXiv  Detail & Related papers  (2026-03-04T10:48:40Z)
• ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning [67.22219034602514]
  We introduce ADEPT (Agentic Decoding of Emotion via Evidence Probing Tools), a framework that reframes emotion recognition as a multi-turn inquiry process.<n> ADEPT transforms an SLLM into an agent that maintains an evolving candidate emotion set and adaptively invokes dedicated semantic and acoustic probing tools.<n>We show that ADEPT improves primary emotion accuracy in most settings while substantially improving minor emotion characterization.
  arXiv  Detail & Related papers  (2026-02-13T08:33:37Z)
• CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering [25.10244503397448]
  Emotional expression in human speech is nuanced and compositional, often involving multiple, conflicting, affective cues.<n>Most expressive text-to-speech systems enforce a single utterance-level emotion, collapsing affective diversity and suppressing mixed or text-emotion-misaligned expression.<n>This paper introduces a quantitative, controllable steering framework, and multi-rater evaluation protocols that enable composable mixed-emotion synthesis and reliable text-emotion mismatch synthesis.
  arXiv  Detail & Related papers  (2026-02-03T11:45:00Z)
• EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis [36.831497786147864]
  We present EmoShift, a lightweight activation-steering framework incorporating a EmoSteer layer.<n>EmoShift learns a steering vector for each target emotion in the output embedding space to capture its latent offset and maintain stable, appropriate expression.<n>With only 10M trainable parameters,less than 1/30 of full fine-tuning, EmoShift outperforms zero-shot and fully fine-tuned baselines in objective and subjective evaluations.
  arXiv  Detail & Related papers  (2026-01-30T11:50:23Z)
• Text-guided Weakly Supervised Framework for Dynamic Facial Expression Recognition [49.41688891301643]
  Dynamic facial expression recognition aims to identify emotional states by modeling the temporal changes in facial movements across video sequences.<n>A key challenge in DFER is the many-to-one labeling problem, where a video composed of numerous frames is assigned a single emotion label.<n>We propose TG-DFER, a text-guided weakly supervised framework that enhances MIL-based DFER by incorporating semantic guidance and coherent temporal modeling.
  arXiv  Detail & Related papers  (2025-11-14T04:49:58Z)
• UDDETTS: Unifying Discrete and Dimensional Emotions for Controllable Emotional Text-to-Speech [61.989360995528905]
  We propose UDDETTS, a universal framework unifying discrete and dimensional emotions for controllable emotional TTS.<n>This model introduces the interpretable Arousal-Dominance-Valence (ADV) space for dimensional emotion description and supports emotion control driven by either discrete emotion labels or nonlinearly quantified ADV values.<n>Experiments show that UDDETTS achieves linear emotion control along three interpretable dimensions, and exhibits superior end-to-end emotional speech synthesis capabilities.
  arXiv  Detail & Related papers  (2025-05-15T12:57:19Z)
• GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations [35.63053777817013]
  GatedxLSTM is a novel multimodal Emotion Recognition in Conversation (ERC) model.<n>It considers voice and transcripts of both the speaker and their conversational partner to identify the most influential sentences driving emotional shifts.<n>It achieves state-of-the-art (SOTA) performance among open-source methods in four-class emotion classification.
  arXiv  Detail & Related papers  (2025-03-26T18:46:18Z)
• UMETTS: A Unified Framework for Emotional Text-to-Speech Synthesis with Multimodal Prompts [64.02363948840333]
  UMETTS is a novel framework that leverages emotional cues from multiple modalities to generate highly expressive and emotionally resonant speech.<n>EP-Align employs contrastive learning to align emotional features across text, audio, and visual modalities, ensuring a coherent fusion of multimodal information.<n>EMI-TTS integrates the aligned emotional embeddings with state-of-the-art TTS models to synthesize speech that accurately reflects the intended emotions.
  arXiv  Detail & Related papers  (2024-04-29T03:19:39Z)
• Dynamic Causal Disentanglement Model for Dialogue Emotion Detection [77.96255121683011]
  We propose a Dynamic Causal Disentanglement Model based on hidden variable separation. This model effectively decomposes the content of dialogues and investigates the temporal accumulation of emotions. Specifically, we propose a dynamic temporal disentanglement model to infer the propagation of utterances and hidden variables.
  arXiv  Detail & Related papers  (2023-09-13T12:58:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.