EmoMind: Decoding Affective Captions from Human Brain fMRI

• URL: http://arxiv.org/abs/2605.16739v1
• Date: Sat, 16 May 2026 01:32:45 GMT
• Title: EmoMind: Decoding Affective Captions from Human Brain fMRI
• Abstract summary: We present EmoMind, the first end-to-end pipeline for decoding affective captions directly from fMRI signals.<n>EmoMind first retrieves a semanti- cally grounded neutral scene description from brain-decoded visual features, then rewrites it using a continuous 34-dimensional emotion vector decoded from the same fMRI recording.
• Score: 6.129853202144421
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decoding visual experience from brain activity has advanced substantially, but cur- rent brain-to-text systems largely recover semantic content while discarding affect. Additionally, language models can generate emotional text when prompted with categorical labels, but such labels collapse rich inter-subject variability into coarse discrete bins. We present EmoMind, the first end-to-end pipeline for decoding affective captions directly from fMRI signals. EmoMind first retrieves a semanti- cally grounded neutral scene description from brain-decoded visual features, then rewrites it using a continuous 34-dimensional emotion vector decoded from the same fMRI recording. To control the balance between content preservation and affective expression, we train the rewriter with classifier-free guidance against an identity-preserving null branch, enabling smooth interpolation between semantic fidelity and affective expressivity. We evaluate affective caption generation with a three-axis validation framework spanning subject-specificity, structural geometry, and causal control. We further augment this framework with a synthetic-brain substitution test that probes robustness to the measurement apparatus, and we benchmark each axis against GPT-4 prompted with brain-decoded top-5 emotion labels as a strong discrete baseline. Across two independent emotion fMRI datasets, EmoMind significantly outperforms label-prompted GPT-4 on all three axes, with the largest gains on metrics that require person-specific affective structure rather than population-level emotion aggregation. These results establish continuous brain-decoded affect as a viable control signal for individualized affective cap- tion generation and open new directions for studying individual affective brain organisation.

Related papers

• A Comparative Study on Affective Cues in Text Embeddings Across Psychological Emotion Theories [6.880856300506082]
  It remains unclear to what extent the latent representations produced by modern text encoders capture well-defined psychological theories of affect.<n>In this work, we investigate the affective capabilities of twelve recently released text encoders by probing their generated embeddings as input features for solving regression and classification tasks.
  arXiv  Detail & Related papers  (2026-06-27T20:09:05Z)
• Navigating the Emotion Tree: Hierarchical Hyperbolic RAG for Multimodal Emotion Recognition [56.650820426231284]
  Multimodal emotion recognition aims to integrate text, audio, and video sources to understand human affective states.<n>Although large language models excel at multimodal reasoning, they typically treat emotion categories as independent labels.<n>We propose bftextHyperEmo-RAG, a retrieval-augmented generation framework that leverages a structured emotional knowledge base.
  arXiv  Detail & Related papers  (2026-05-16T10:04:48Z)
• Memory-guided Prototypical Co-occurrence Learning for Mixed Emotion Recognition [56.00118641432005]
  We propose a Memory-guided Prototypical Co-occurrence Learning framework that explicitly models emotion co-occurrence patterns.<n>Inspired by human cognitive memory systems, we introduce a memory retrieval strategy to extract semantic-level co-occurrence associations.<n>Our model learns affectively informative representations for accurate emotion distribution prediction.
  arXiv  Detail & Related papers  (2026-02-24T04:11:25Z)
• ADEPT: RL-Aligned Agentic Decoding of Emotion via Evidence Probing Tools -- From Consensus Learning to Ambiguity-Driven Emotion Reasoning [67.22219034602514]
  We introduce ADEPT (Agentic Decoding of Emotion via Evidence Probing Tools), a framework that reframes emotion recognition as a multi-turn inquiry process.<n> ADEPT transforms an SLLM into an agent that maintains an evolving candidate emotion set and adaptively invokes dedicated semantic and acoustic probing tools.<n>We show that ADEPT improves primary emotion accuracy in most settings while substantially improving minor emotion characterization.
  arXiv  Detail & Related papers  (2026-02-13T08:33:37Z)
• Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models [8.550786156000461]
  We present the first neuron-level interpretability study of emotion-sensitive neurons (ESNs) in large audio-language models (LALMs)<n>We compare frequency-, entropy-, magnitude-, and contrast-based neuron selectors on multiple emotion recognition benchmarks.<n>Using inference-time interventions, we reveal a consistent emotion-specific signature.
  arXiv  Detail & Related papers  (2026-01-06T15:46:35Z)
• DepFlow: Disentangled Speech Generation to Mitigate Semantic Bias in Depression Detection [54.209716321122194]
  We present DepFlow, a depression-conditioned text-to-speech framework.<n>A Depression Acoustic Camouflage learns speaker- and content-invariant depression embeddings through adversarial training.<n>A flow-matching TTS model with FiLM modulation injects these embeddings into synthesis, enabling control over depressive severity.<n>A prototype-based severity mapping mechanism provides smooth and interpretable manipulation across the depression continuum.
  arXiv  Detail & Related papers  (2026-01-01T10:44:38Z)
• A Brain Wave Encodes a Thousand Tokens: Modeling Inter-Cortical Neural Interactions for Effective EEG-based Emotion Recognition [0.41998444721319217]
  We propose RBTransformer, a Transformer-based neural network architecture that models inter-cortical neural dynamics of the brain in latent space.<n>We conducted extensive experiments, specifically under subject-dependent settings, on the SEED, DEAP, and DREAMER datasets.<n>The results demonstrate that the proposed RBTransformer outperforms all previous state-of-the-art methods across all three datasets.
  arXiv  Detail & Related papers  (2025-11-17T22:27:12Z)
• Decoding Neural Emotion Patterns through Large Language Model Embeddings [3.8032942955371785]
  We propose a computational framework that maps textual emotional content to anatomically defined brain regions without requiring neuroimaging.<n>Using OpenAI's text-embedding-ada-ada, we generate high-dimensional semantic representations, apply dimensionality reduction and clustering to identify emotional groups, and map them to 18 brain regions linked to emotional processing.<n>This cost-effective, scalable approach enables large-scale analysis of naturalistic language, distinguishes between clinical populations, and offers a brain-based benchmark for evaluating AI emotional expression.
  arXiv  Detail & Related papers  (2025-08-12T20:51:56Z)
• Learning Interpretable Representations Leads to Semantically Faithful EEG-to-Text Generation [52.51005875755718]
  We focus on EEG-to-text decoding and address its hallucination issue through the lens of posterior collapse.<n>Acknowledging the underlying mismatch in information capacity between EEG and text, we reframe the decoding task as semantic summarization of core meanings.<n>Experiments on the public ZuCo dataset demonstrate that GLIM consistently generates fluent, EEG-grounded sentences.
  arXiv  Detail & Related papers  (2025-05-21T05:29:55Z)
• learning discriminative features from spectrograms using center loss for speech emotion recognition [62.13177498013144]
  We propose a novel approach to learn discriminative features from variable length spectrograms for emotion recognition.<n>The softmax cross-entropy loss enables features from different emotion categories separable, and the center loss efficiently pulls the features belonging to the same emotion category to their center.
  arXiv  Detail & Related papers  (2025-01-02T06:52:28Z)
• BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation [48.20672677492805]
  Current EEG/MEG-to-text decoding systems suffer from three key limitations.<n>BrainECHO is a multi-stage framework that employs decoupled representation learning.<n>BrainECHO demonstrates robustness across sentence, session, and subject-independent conditions.
  arXiv  Detail & Related papers  (2024-10-19T04:29:03Z)
• Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping [55.72376663488104]
  We present an autoencoder-based approach to classify perceived human emotions from walking styles obtained from videos or motion-captured data. Given the motion on each joint in the pose at each time step extracted from 3D pose sequences, we hierarchically pool these joint motions in the encoder. We train the decoder to reconstruct the motions per joint per time step in a top-down manner from the latent embeddings.
  arXiv  Detail & Related papers  (2019-11-20T05:04:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.