Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation

• URL: http://arxiv.org/abs/2608.22399v1
• Date: Sun, 23 Aug 2026 12:56:03 GMT
• Title: Dual-Scale State-Space Modeling with Speaker-Wise Dynamic CRF for Speech Emotion Recognition in Conversation
• Abstract summary: We propose DSSM-CRF, an audio-only architecture for emotion recognition.<n>Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales.<n> Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances.
• Score: 1.6389263804357965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational speech emotion recognition must reconcile acoustic evidence across temporal scales with two interaction processes: cross-speaker contextual influence and within-speaker emotion evolution. We propose DSSM-CRF, an audio-only architecture that explicitly separates these processes. Bidirectional state-space models encode fused self-supervised speech representations at frame and dialogue scales, so each utterance representation captures local prosody and context from all speakers. The decoder then orders each speaker's utterances into an independent dynamic conditional random field chain. Consecutive utterances in a speaker's chain form a transition pair whose score combines a corpus-level transition matrix with a residual predicted from the two contextualized utterances. An auxiliary objective supervises whether each pair changes emotion but does not participate in Viterbi inference. Thus, interlocutor turns affect contextual emotion scores without being treated as transitions in another speaker's emotion trajectory. DSSM-CRF achieves 75.81% UA and 74.90% WA on IEMOCAP, and 54.72% WA and 49.31% WF1 on MELD. Matched controls demonstrate complementary gains from speaker-wise factorization and CRF modeling.

Related papers

• KABURI-TTS: Phoneme-Keyed Activity-conditioned Bi-channel Utterance Rendering for Interaction [29.52851418305386]
  KABURITTS takes a per-speaker phoneme as input and renders the speech of two speakers on per-frame phonemes.<n>A user evaluation shows that, compared with strong baselines, the proposed method attains naturalness at both utterance and the interaction level.
  arXiv  Detail & Related papers  (2026-09-07T08:24:10Z)
• UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction [7.775050285048427]
  Speech assistant is driving artificial propagation toward human-like conversational systems.<n>Front-end components, such as voice activity detection (VAD) and turn-taking detection (TD) are crucial for speech assistant.<n>We propose the first unified audio frontend LLM (UAF) tailored for fullgressive speech systems.
  arXiv  Detail & Related papers  (2026-04-21T08:24:55Z)
• Disentangled Dual-Branch Graph Learning for Conversational Emotion Recognition [37.30058027224552]
  Multimodal emotion recognition in conversations aims to infer utterance-level emotions by jointly modeling textual, acoustic, and visual cues within context.<n>We propose a framework that combines dual-space feature disentanglement with dual-branch graph learning.<n> Experiments on IEMOCAP and MELD demonstrate that the proposed method achieves superior performance over strong baselines.
  arXiv  Detail & Related papers  (2026-04-03T14:47:26Z)
• Disentangling Voice and Content with Self-Supervision for Speaker Recognition [57.446013973449645]
  This paper proposes a disentanglement framework that simultaneously models speaker traits and content variability in speech. It is validated with experiments conducted on the VoxCeleb and SITW datasets with 9.56% and 8.24% average reductions in EER and minDCF.
  arXiv  Detail & Related papers  (2023-10-02T12:02:07Z)
• Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation and Recognition [52.11964238935099]
  An audio-visual multi-channel speech separation, dereverberation and recognition approach is proposed in this paper. Video input is consistently demonstrated in mask-based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset.
  arXiv  Detail & Related papers  (2023-07-06T10:50:46Z)
• deep learning of segment-level feature representation for speech emotion recognition in conversations [9.432208348863336]
  We propose a conversational speech emotion recognition method to deal with capturing attentive contextual dependency and speaker-sensitive interactions. First, we use a pretrained VGGish model to extract segment-based audio representation in individual utterances. Second, an attentive bi-directional recurrent unit (GRU) models contextual-sensitive information and explores intra- and inter-speaker dependencies jointly.
  arXiv  Detail & Related papers  (2023-02-05T16:15:46Z)
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
  One-shot voice conversion can be effectively achieved by speech representation disentanglement. We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
  arXiv  Detail & Related papers  (2021-06-18T13:50:38Z)
• A Hierarchical Transformer with Speaker Modeling for Emotion Recognition in Conversation [12.065178204539693]
  Emotion Recognition in Conversation (ERC) is a personalized and interactive emotion recognition task. Current method models speakers' interactions by building a relation between every two speakers. We simplify the complicated modeling to a binary version: Intra-Speaker and Inter-Speaker dependencies.
  arXiv  Detail & Related papers  (2020-12-29T14:47:35Z)
• FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention [66.77490220410249]
  We propose FragmentVC, in which the latent phonetic structure of the utterance from the source speaker is obtained from Wav2Vec 2.0. FragmentVC is able to extract fine-grained voice fragments from the target speaker utterance(s) and fuse them into the desired utterance. This approach is trained with reconstruction loss only without any disentanglement considerations between content and speaker information.
  arXiv  Detail & Related papers  (2020-10-27T09:21:03Z)
• Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue [76.88174667929665]
  A multi-turn dialogue is composed of multiple utterances from two or more different speaker roles. In the existing retrieval-based multi-turn dialogue modeling, the pre-trained language models (PrLMs) as encoder represent the dialogues coarsely. We propose a novel model to fill such a gap by modeling the effective utterance-aware and speaker-aware representations entailed in a dialogue history.
  arXiv  Detail & Related papers  (2020-09-14T15:07:19Z)
• Converting Anyone's Emotion: Towards Speaker-Independent Emotional Voice Conversion [83.14445041096523]
  Emotional voice conversion aims to convert the emotion of speech from one state to another while preserving the linguistic content and speaker identity. We propose a speaker-independent emotional voice conversion framework, that can convert anyone's emotion without the need for parallel data. Experiments show that the proposed speaker-independent framework achieves competitive results for both seen and unseen speakers.
  arXiv  Detail & Related papers  (2020-05-13T13:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.