Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition

• URL: http://arxiv.org/abs/2609.05755v1
• Date: Fri, 04 Sep 2026 22:34:14 GMT
• Title: Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition
• Abstract summary: multimodal recognizers collapse human emotion onto a single hard label.<n>We argue the recognizer should instead expose a distribution over affective space.<n>Our system emits a $9times9$ probability matrix over the Valence-Arousal plane.
• Score: 0.4779196219827507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human emotion is graded and frequently mixed, yet most multimodal recognizers collapse it onto a single hard label. We argue the recognizer should instead expose a distribution over affective space. Our text+speech system, alongside its categorical decision, emits a $9\times9$ probability matrix over the Valence-Arousal plane, trained with a two-dimensional Gaussian soft target under a Kullback-Leibler/cross-entropy objective, aimed at counseling support. Evaluation is strict: speaker-independent 5-fold leave-one-session-out IEMOCAP with rotating-session inner validation, headline metrics only on the held-out session. Within one fixed encoder-fusion-head pipeline we compare Transformer and state-space (Mamba-1/2/3) backbones at matched depth and width, at two operating points ($T\approx550$, $T\approx2750$). The featured dual-head system reaches 73.0% $\pm$ 0.3 unweighted accuracy over three seeds (separate rerun: 72.1%), exceeding the Transformer fusion baseline by 3.0 UA points (95% session-bootstrap CI [1.0,4.7]; significant under paired t-test and session-level bootstrap), with no latency or memory advantage at these lengths; swapping the ~1M trainable front-end for frozen WavLM-Large features (learnable layer weights) lifts the same architecture to 76.6% $\pm$ 1.3. Pre-specified controls scope the claims honestly: simpler valence-arousal auxiliaries reproduce the classification lift within noise, and a dedicated regression head tracks the continuous ratings slightly better, so the head's specific value is the normalized affect distribution itself. That distribution recovers the circumplex: its center of mass tracks valence and arousal (CCC 0.66/0.66; predominantly between-class structure, weaker within-class tracking), and its entropy is weakly but consistently linked to categorical rater ambiguity, not dimensional spread.

Related papers

• Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs [43.01574038234673]
  Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count.<n>We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm.
  arXiv  Detail & Related papers  (2026-09-10T16:17:19Z)
• Round-Trip Consistency: Bidirectional Diffusion Models Can Predict Their Own Rollout Errors [51.56484100374058]
  Autoregressive models accumulate error over long rollouts, yet at deployment there is no ground truth to measure it against.<n>We train a single conditional latent diffusion model that steps a dynamical system forward or backward in time via a direction flag.<n>We show that this bidirectionality supplies a measurement-free test-time error signal.
  arXiv  Detail & Related papers  (2026-08-01T13:49:46Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics [0.0]
  Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse.<n>We show that weight decay acts as a scalar empirical control parameter for these regimes.<n>We introduce online diagnostics, mean pairwise attentionhead cosine similarity and entropy standard deviation, that track training dynamics.
  arXiv  Detail & Related papers  (2026-05-19T19:48:40Z)
• Scale-Equivariant Generative Forecasting: Weight-Tied Dilated Convolutions, Wavelet Scattering Inputs, and Spectral-Consistency Training for Self-Similar Time Series [0.0]
  We give a precise definition of discrete scale equivariance for 1D causal networks.<n>We prove that dyadic dilation commutes with any dilated-convolution stack whose kernel weights are shared across levels.<n>On 30 years of S&P 500 daily log-returns, SE-WaveNet samples reproduce the empirical scaling-collapse diagnostic on the Allan-Variance top-25 universe.
  arXiv  Detail & Related papers  (2026-05-17T18:21:30Z)
• Why Training-Free Token Reduction Collapses: The Inherent Instability of Pairwise Scoring Signals [0.0]
  Training-free token reduction methods for Vision Transformers employ different scoring mechanisms, yet they share a closely matched cliff-like collapse at high compression.<n>We develop a diagnostic framework with two tools, ranking consistency $_s$ and off-diagonal correlation $_textoff$, that decomposes the collapse into (1)a signal-agnostic error amplifier inherent to layer-wise reduction.<n>We construct CATIS as a constructive validation: unary signals raise the trigger threshold, triage suppresses the gain.
  arXiv  Detail & Related papers  (2026-04-17T23:26:27Z)
• Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold Steering [0.0]
  Quantized language models face a fundamental dilemma: low sampling temperatures yield repetitive, mode-collapsed outputs, while high temperatures (T > 2.0) cause trajectory divergence and semantic incoherence.<n>We present HELIX, a framework that decouples output entropy from hallucination by tethering hidden-state trajectories to a pre-computed truthfulness manifold.
  arXiv  Detail & Related papers  (2026-02-06T06:24:37Z)
• Phase Transition for Budgeted Multi-Agent Synergy [41.486076708302456]
  Multi-agent systems can improve reliability, yet under a fixed inference budget they often help, saturate, or even collapse.<n>We develop a minimal and calibratable theory that predicts these regimes from three binding constraints of modern agent stacks.
  arXiv  Detail & Related papers  (2026-01-24T05:32:50Z)
• Robust inverse material design with physical guarantees using the Voigt-Reuss Net [0.0]
  We propose a spectrally normalized surrogate for forward and inverse mechanical homogenization with hard physical guarantees.<n>In 3D linear elasticity on an open dataset of biphasic microstructures, a fully connected Voigt-Reuss net trained on $>!7.5times 105$ FFT-based labels with 236 isotropy-in descriptors.<n>Overall, the Voigt-Reuss net unifies accurate, physically admissible forward prediction with large-batch, constraint-consistent inverse design.
  arXiv  Detail & Related papers  (2025-11-14T15:17:37Z)
• Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping [42.2819343711085]
  We show that FL with DP is viable under strong privacy guarantees, provided a population of at least several million users.<n>We achieve user-level (7.2, $10-9$)-DP (resp. (4.5, $10-9$)-DP) with only a 1.3% absolute drop in word error rate when extrapolating to high (resp. low) population scales for FL with DP in ASR.
  arXiv  Detail & Related papers  (2023-09-29T19:11:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.