MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models

• URL: http://arxiv.org/abs/2605.14635v1
• Date: Thu, 14 May 2026 09:49:32 GMT
• Title: MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
• Abstract summary: We introduce a new benchmark dataset for visual emotion analysis.<n>We hire $20$ annotators per image and ask them to select all emotions they feel from an image.<n>The resulting dataset contains $10,344$ images with $236,998$ valid votes across eight emotions.
• Score: 6.924490096929709
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper introduces a multi-label visual emotion analysis benchmark dataset for comprehensively evaluating the ability of multimodal large language models (MLLMs) to predict the emotions evoked by images. Recent user studies report an unintuitive finding: humans may prefer the predictions of MLLMs over the labels in existing datasets. We argue that this phenomenon stems from the suboptimal annotation scheme used in existing datasets, where each annotator is shown a single candidate emotion for each image and judges whether it is evoked or not. This approach is clearly limited because a single image can evoke multiple emotions with varying intensities. As a result, evaluations based on these datasets may underestimate the capabilities of MLLMs, yet an appropriate benchmark for evaluating such models remains lacking. To address this issue, we introduce a new multi-label benchmark dataset for visual emotion analysis toward MLLMs evaluation. We hire $20$ annotators per image and ask them to select all emotions they feel from an image. Then, we aggregate the votes across all annotators, providing a more reliable and representative dataset labeled with a distribution of emotions. The resulting dataset contains $10,344$ images with $236,998$ valid votes across eight emotions. Based on this benchmark dataset, we evaluate several recent models, including Qwen3-VL, OpenAI's GPT, Gemini, and Claude. We assess model performance on both dominant emotion prediction and emotion distribution prediction. Our results demonstrate the progress achieved by recent MLLMs while also indicating that substantial room for improvement remains. Furthermore, our experiments with LLM-as-a-judge show that the method does not consistently improve MLLMs' performance, indicating its limitations for the subjective task of visual emotion analysis.

Related papers

• Do Multimodal LLMs See Sentiment? [1.990154090209513]
  We propose an original framework, MLLMsent, to investigate the sentiment reasoning capabilities of Multimodal Large Language Models (MLLMs)<n> Experiments on a recent and established benchmark demonstrate that our proposal achieves, state-of-the-art results.
  arXiv  Detail & Related papers  (2025-08-23T02:11:46Z)
• MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models [108.61337743051483]
  We present MME-Emotion, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs.<n>MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks.<n>It incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework.
  arXiv  Detail & Related papers  (2025-08-11T03:14:55Z)
• Evaluating Graphical Perception with Multimodal LLMs [2.090547583226381]
  Multimodal Large Language Models (MLLMs) have remarkably progressed in analyzing and understanding images.<n>For visualization, how do MLLMs perform when applied to graphical perception tasks?<n>Our study primarily evaluates fine-tuned and pretrained models and zero-shot prompting to determine if they closely match human graphical perception.
  arXiv  Detail & Related papers  (2025-04-05T16:14:08Z)
• VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning [63.0285363282581]
  Multimodal Large Language Models (MLLMs) have become a powerful tool for integrating visual and textual information.<n>We introduce VOILA, a benchmark designed to evaluate MLLMs' perceptual understanding and abstract relational reasoning.<n>We reveal that current MLLMs struggle to comprehend inter-image relationships and exhibit limited capabilities in high-level relational reasoning.
  arXiv  Detail & Related papers  (2025-02-25T23:36:19Z)
• EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents [57.468696197956604]
  EmbodiedEval is a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks.<n>It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity.<n>We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks.
  arXiv  Detail & Related papers  (2025-01-21T03:22:10Z)
• Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis [6.387263468033964]
  We introduce a self-reviewed dataset and a human-reviewed dataset, comprising 24,137 coarse-grained samples and 3,500 manually annotated samples with detailed emotion annotations.<n>In addition to the audio modeling, we propose to explicitly integrate facial encoding models into the existing advanced Video MLLM.<n>Our Omni-Emotion achieves state-of-the-art performance in both emotion recognition and reasoning tasks.
  arXiv  Detail & Related papers  (2025-01-16T12:27:05Z)
• MEMO-Bench: A Multiple Benchmark for Text-to-Image and Multimodal Large Language Models on Human Emotion Analysis [53.012111671763776]
  This study introduces MEMO-Bench, a comprehensive benchmark consisting of 7,145 portraits, each depicting one of six different emotions. Results demonstrate that existing T2I models are more effective at generating positive emotions than negative ones. Although MLLMs show a certain degree of effectiveness in distinguishing and recognizing human emotions, they fall short of human-level accuracy.
  arXiv  Detail & Related papers  (2024-11-18T02:09:48Z)
• LIME: Less Is More for MLLM Evaluation [36.29820380945517]
  We propose LIME (Less Is More for MLLM Evaluation), a benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models.
  arXiv  Detail & Related papers  (2024-09-10T20:19:14Z)
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning [55.127202990679976]
  We introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. We propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders.
  arXiv  Detail & Related papers  (2024-06-17T03:01:22Z)
• Q-Bench+: A Benchmark for Multi-modal Foundation Models on Low-level Vision from Single Images to Pairs [71.07108539262721]
  We design benchmark settings to emulate human language responses related to low-level vision. We extend the low-level perception-related question-answering and description evaluations of MLLMs from single images to image pairs. We demonstrate that several MLLMs have decent low-level visual competencies on single images, but only GPT-4V exhibits higher accuracy on pairwise comparisons than humans.
  arXiv  Detail & Related papers  (2024-02-11T06:44:11Z)
• The Instinctive Bias: Spurious Images lead to Illusion in MLLMs [34.91795817316696]
  We identify a typical class of inputs that baffles MLLMs, which consist of images that are highly relevant but inconsistent with answers. We propose CorrelationQA, the first benchmark that assesses the visual illusion level given spurious images. We conduct a thorough analysis on 9 mainstream MLLMs, illustrating that they universally suffer from this instinctive bias to varying degrees.
  arXiv  Detail & Related papers  (2024-02-06T06:48:46Z)
• Reevaluating Data Partitioning for Emotion Detection in EmoWOZ [0.0]
  EmoWoz is an extension of MultiWOZ that provides emotion labels for the dialogues. MultiWOZ was partitioned initially for another purpose, resulting in a distributional shift when considering the new purpose of emotion recognition. We propose a stratified sampling scheme based on emotion tags to address this issue, improve the dataset's distribution, and reduce dataset shift.
  arXiv  Detail & Related papers  (2023-03-15T03:06:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.