EmoGist: Efficient In-Context Learning for Visual Emotion Understanding

• URL: http://arxiv.org/abs/2505.14660v1
• Date: Tue, 20 May 2025 17:47:04 GMT
• Title: EmoGist: Efficient In-Context Learning for Visual Emotion Understanding
• Authors: Ronald Seoh, Dan Goldwasser,
• Abstract summary: EmoGist is a training-free, in-context learning method for performing visual emotion classification with LVLMs.<n>We show that EmoGist allows up to 13 points improvement in micro F1 scores with the multi-label Memotion dataset.
• Score: 19.979621982792885
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we introduce EmoGist, a training-free, in-context learning method for performing visual emotion classification with LVLMs. The key intuition of our approach is that context-dependent definition of emotion labels could allow more accurate predictions of emotions, as the ways in which emotions manifest within images are highly context dependent and nuanced. EmoGist pre-generates multiple explanations of emotion labels, by analyzing the clusters of example images belonging to each category. At test time, we retrieve a version of explanation based on embedding similarity, and feed it to a fast VLM for classification. Through our experiments, we show that EmoGist allows up to 13 points improvement in micro F1 scores with the multi-label Memotion dataset, and up to 8 points in macro F1 in the multi-class FI dataset.

Related papers

• Empaths at SemEval-2025 Task 11: Retrieval-Augmented Approach to Perceived Emotions Prediction [83.88591755871734]
  EmoRAG is a system designed to detect perceived emotions in text for SemEval-2025 Task 11, Subtask A: Multi-label Emotion Detection.<n>We focus on predicting the perceived emotions of the speaker from a given text snippet, labeling it with emotions such as joy, sadness, fear, anger, surprise, and disgust.
  arXiv  Detail & Related papers  (2025-06-04T19:41:24Z)
• Emo Pillars: Knowledge Distillation to Support Fine-Grained Context-Aware and Context-Less Emotion Classification [56.974545305472304]
  Most datasets for sentiment analysis lack context in which an opinion was expressed, often crucial for emotion understanding, and are mainly limited by a few emotion categories.<n>We design an LLM-based data synthesis pipeline and leverage a large model, Mistral-7b, for the generation of training examples for more accessible, lightweight BERT-type encoder models.<n>We show that Emo Pillars models are highly adaptive to new domains when tuned to specific tasks such as GoEmotions, ISEAR, IEMOCAP, and EmoContext, reaching the SOTA performance on the first three.
  arXiv  Detail & Related papers  (2025-04-23T16:23:17Z)
• EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes [53.95428298229396]
  We introduce EmoSet, the first large-scale visual emotion dataset annotated with rich attributes. EmoSet comprises 3.3 million images in total, with 118,102 of these images carefully labeled by human annotators. Motivated by psychological studies, in addition to emotion category, each image is also annotated with a set of describable emotion attributes.
  arXiv  Detail & Related papers  (2023-07-16T06:42:46Z)
• Leveraging Label Correlations in a Multi-label Setting: A Case Study in Emotion [0.0]
  We exploit label correlations in multi-label emotion recognition models to improve emotion detection. We demonstrate state-of-the-art performance across Spanish, English, and Arabic in SemEval 2018 Task 1 E-c using monolingual BERT-based models.
  arXiv  Detail & Related papers  (2022-10-28T02:27:18Z)
• VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition [21.247650660908484]
  This paper proposes a multimodal emotion recognition system, VIsual Textual Additive Net (VISTANet) The VISTANet fuses information from image, speech, and text modalities using a hybrid of early and late fusion. The KAAP technique computes the contribution of each modality and corresponding features toward predicting a particular emotion class.
  arXiv  Detail & Related papers  (2022-08-24T11:35:51Z)
• A Circular-Structured Representation for Visual Emotion Distribution Learning [82.89776298753661]
  We propose a well-grounded circular-structured representation to utilize the prior knowledge for visual emotion distribution learning. To be specific, we first construct an Emotion Circle to unify any emotional state within it. On the proposed Emotion Circle, each emotion distribution is represented with an emotion vector, which is defined with three attributes.
  arXiv  Detail & Related papers  (2021-06-23T14:53:27Z)
• Enhancing Cognitive Models of Emotions with Representation Learning [58.2386408470585]
  We present a novel deep learning-based framework to generate embedding representations of fine-grained emotions. Our framework integrates a contextualized embedding encoder with a multi-head probing model. Our model is evaluated on the Empathetic Dialogue dataset and shows the state-of-the-art result for classifying 32 emotions.
  arXiv  Detail & Related papers  (2021-04-20T16:55:15Z)
• SpanEmo: Casting Multi-label Emotion Classification as Span-prediction [15.41237087996244]
  We propose a new model "SpanEmo" casting multi-label emotion classification as span-prediction. We introduce a loss function focused on modelling multiple co-existing emotions in the input sentence. Experiments performed on the SemEval2018 multi-label emotion data over three language sets demonstrate our method's effectiveness.
  arXiv  Detail & Related papers  (2021-01-25T12:11:04Z)
• EmoGraph: Capturing Emotion Correlations using Graph Networks [71.53159402053392]
  We propose EmoGraph that captures the dependencies among different emotions through graph networks. EmoGraph outperforms strong baselines, especially for macro-F1. An experiment illustrates the captured emotion correlations can also benefit a single-label classification task.
  arXiv  Detail & Related papers  (2020-08-21T08:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.