EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding

• URL: http://arxiv.org/abs/2412.08049v3
• Date: Mon, 31 Mar 2025 07:15:17 GMT
• Title: EmoVerse: Exploring Multimodal Large Language Models for Sentiment and Emotion Understanding
• Authors: Ao Li, Longwei Xu, Chen Ling, Jinghui Zhang, Pengwei Wang,
• Abstract summary: We introduce Emotion Universe (EmoVerse), an MLLM designed to handle a broad spectrum of sentiment and emotion-related tasks.<n>EmoVerse is capable of deeply analyzing the underlying causes of emotional states.<n>We also introduce the Affective Multitask (AMT) dataset.
• Score: 5.3848462080869215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sentiment and emotion understanding are essential to applications such as human-computer interaction and depression detection. While Multimodal Large Language Models (MLLMs) demonstrate robust general capabilities, they face considerable challenges in the field of affective computing, particularly in detecting subtle facial expressions and handling complex emotion-related tasks, such as emotion reason inference and understanding emotions in long-context scenarios. Furthermore, there is a lack of a unified MLLM that can effectively handle both sentiment and emotion-related tasks. To address these challenges, we explore multi-task training strategies for MLLMs in affective computing and introduce Emotion Universe (EmoVerse), an MLLM designed to handle a broad spectrum of sentiment and emotion-related tasks. In addition, EmoVerse is capable of deeply analyzing the underlying causes of emotional states. We also introduce the Affective Multitask (AMT) Dataset, which supports multimodal sentiment analysis, multimodal emotion recognition, facial expression recognition, emotion reason inference, and emotion cause-pair extraction tasks. Extensive experiments demonstrate that EmoVerse outperforms existing methods, achieving state-of-the-art results in sentiment and emotion-related tasks. The code is available at https://github.com/liaolea/EmoVerse.

Related papers

• AI with Emotions: Exploring Emotional Expressions in Large Language Models [0.0]
  Large Language Models (LLMs) play role-play as agents answering questions with specified emotional states. Russell's Circumplex model characterizes emotions along the sleepy-activated (arousal) and pleasure-displeasure (valence) axes. evaluation showed that the emotional states of the generated answers were consistent with the specifications.
  arXiv  Detail & Related papers  (2025-04-20T18:49:25Z)
• MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis [53.012111671763776]
  This study introduces MEMO-Bench, a comprehensive benchmark consisting of 7,145 portraits, each depicting one of six different emotions. Results demonstrate that existing T2I models are more effective at generating positive emotions than negative ones. Although MLLMs show a certain degree of effectiveness in distinguishing and recognizing human emotions, they fall short of human-level accuracy.
  arXiv  Detail & Related papers  (2024-11-18T02:09:48Z)
• AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models [18.482881562645264]
  This study is the first to explore the potential of Large Language Models (LLMs) in recognizing ambiguous emotions. We design zero-shot and few-shot prompting and incorporate past dialogue as context information for ambiguous emotion recognition.
  arXiv  Detail & Related papers  (2024-09-26T23:25:21Z)
• EmoLLM: Multimodal Emotional Understanding Meets Large Language Models [61.179731667080326]
  Multi-modal large language models (MLLMs) have achieved remarkable performance on objective multimodal perception tasks. But their ability to interpret subjective, emotionally nuanced multimodal content remains largely unexplored. EmoLLM is a novel model for multimodal emotional understanding, incorporating with two core techniques.
  arXiv  Detail & Related papers  (2024-06-24T08:33:02Z)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning [55.127202990679976]
  We introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. We propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders.
  arXiv  Detail & Related papers  (2024-06-17T03:01:22Z)
• Think out Loud: Emotion Deducing Explanation in Dialogues [57.90554323226896]
  We propose a new task "Emotion Deducing Explanation in Dialogues" (EDEN) EDEN recognizes emotion and causes in an explicitly thinking way. It can help Large Language Models (LLMs) achieve better recognition of emotions and causes.
  arXiv  Detail & Related papers  (2024-06-07T08:58:29Z)
• SemEval-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations [53.60993109543582]
  SemEval-2024 Task 3, named Multimodal Emotion Cause Analysis in Conversations, aims at extracting all pairs of emotions and their corresponding causes from conversations. Under different modality settings, it consists of two subtasks: Textual Emotion-Cause Pair Extraction in Conversations (TECPE) and Multimodal Emotion-Cause Pair Extraction in Conversations (MECPE) In this paper, we introduce the task, dataset and evaluation settings, summarize the systems of the top teams, and discuss the findings of the participants.
  arXiv  Detail & Related papers  (2024-05-19T09:59:00Z)
• UniMEEC: Towards Unified Multimodal Emotion Recognition and Emotion Cause [18.99103120856208]
  We propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework (UniMEEC) to explore the causality between emotion and emotion cause. UniMEEC reformulates the MERC and MECPE tasks as mask prediction problems and unifies them with a causal prompt template. Experiment results on four public benchmark datasets verify the model performance on MERC and MECPE tasks.
  arXiv  Detail & Related papers  (2024-03-30T15:59:17Z)
• Enhancing Emotional Generation Capability of Large Language Models via Emotional Chain-of-Thought [50.13429055093534]
  Large Language Models (LLMs) have shown remarkable performance in various emotion recognition tasks. We propose the Emotional Chain-of-Thought (ECoT) to enhance the performance of LLMs on various emotional generation tasks.
  arXiv  Detail & Related papers  (2024-01-12T16:42:10Z)
• Large Language Models Understand and Can be Enhanced by Emotional Stimuli [53.53886609012119]
  We take the first step towards exploring the ability of Large Language Models to understand emotional stimuli. Our experiments show that LLMs have a grasp of emotional intelligence, and their performance can be improved with emotional prompts. Our human study results demonstrate that EmotionPrompt significantly boosts the performance of generative tasks.
  arXiv  Detail & Related papers  (2023-07-14T00:57:12Z)
• Emotion Recognition from Multiple Modalities: Fundamentals and Methodologies [106.62835060095532]
  We discuss several key aspects of multi-modal emotion recognition (MER) We begin with a brief introduction on widely used emotion representation models and affective modalities. We then summarize existing emotion annotation strategies and corresponding computational tasks. Finally, we outline several real-world applications and discuss some future directions.
  arXiv  Detail & Related papers  (2021-08-18T21:55:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.