Decoding Emotions in Abstract Art: Cognitive Plausibility of CLIP in Recognizing Color-Emotion Associations

• URL: http://arxiv.org/abs/2405.06319v1
• Date: Fri, 10 May 2024 08:45:23 GMT
• Title: Decoding Emotions in Abstract Art: Cognitive Plausibility of CLIP in Recognizing Color-Emotion Associations
• Authors: Hanna-Sophia Widhoelzl, Ece Takmaz,
• Abstract summary: This study investigates the cognitive plausibility of a pretrained multimodal model, CLIP, in recognizing emotions evoked by abstract visual art. We employ a dataset comprising images with associated emotion labels and textual rationales of these labels provided by human annotators. We perform linguistic analyses of rationales, zero-shot emotion classification of images and rationales, apply similarity-based prediction of emotion, and investigate color-emotion associations.
• Score: 1.0659364666674607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study investigates the cognitive plausibility of a pretrained multimodal model, CLIP, in recognizing emotions evoked by abstract visual art. We employ a dataset comprising images with associated emotion labels and textual rationales of these labels provided by human annotators. We perform linguistic analyses of rationales, zero-shot emotion classification of images and rationales, apply similarity-based prediction of emotion, and investigate color-emotion associations. The relatively low, yet above baseline, accuracy in recognizing emotion for abstract images and rationales suggests that CLIP decodes emotional complexities in a manner not well aligned with human cognitive processes. Furthermore, we explore color-emotion interactions in images and rationales. Expected color-emotion associations, such as red relating to anger, are identified in images and texts annotated with emotion labels by both humans and CLIP, with the latter showing even stronger interactions. Our results highlight the disparity between human processing and machine processing when connecting image features and emotions.

Related papers

• Emotional Images: Assessing Emotions in Images and Potential Biases in Generative Models [0.0]
  This paper examines potential biases and inconsistencies in emotional evocation of images produced by generative artificial intelligence (AI) models. We compare the emotions evoked by an AI-produced image to the emotions evoked by prompts used to create those images. Findings indicate that AI-generated images frequently lean toward negative emotional content, regardless of the original prompt.
  arXiv  Detail & Related papers  (2024-11-08T21:42:50Z)
• Learning Emotion Representations from Verbal and Nonverbal Communication [7.747924294389427]
  We present EmotionCLIP, the first pre-training paradigm to extract visual emotion representations from verbal and nonverbal communication. We guide EmotionCLIP to attend to nonverbal emotion cues through subject-aware context encoding and verbal emotion cues using sentiment-guided contrastive learning. EmotionCLIP will address the prevailing issue of data scarcity in emotion understanding, thereby fostering progress in related domains.
  arXiv  Detail & Related papers  (2023-05-22T21:36:55Z)
• SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network [83.27291945217424]
  We propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict emotions from images. To mine the emotional relationships between distinct objects, we first build up an Emotion Graph based on semantic concepts and visual features. We also design a Scene-Object Fusion Module to integrate scenes and objects, which exploits scene features to guide the fusion process of object features with the proposed scene-based attention mechanism.
  arXiv  Detail & Related papers  (2021-10-24T02:41:41Z)
• Stimuli-Aware Visual Emotion Analysis [75.68305830514007]
  We propose a stimuli-aware visual emotion analysis (VEA) method consisting of three stages, namely stimuli selection, feature extraction and emotion prediction. To the best of our knowledge, it is the first time to introduce stimuli selection process into VEA in an end-to-end network. Experiments demonstrate that the proposed method consistently outperforms the state-of-the-art approaches on four public visual emotion datasets.
  arXiv  Detail & Related papers  (2021-09-04T08:14:52Z)
• Affective Image Content Analysis: Two Decades Review and New Perspectives [132.889649256384]
  We will comprehensively review the development of affective image content analysis (AICA) in the recent two decades. We will focus on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
  arXiv  Detail & Related papers  (2021-06-30T15:20:56Z)
• A Circular-Structured Representation for Visual Emotion Distribution Learning [82.89776298753661]
  We propose a well-grounded circular-structured representation to utilize the prior knowledge for visual emotion distribution learning. To be specific, we first construct an Emotion Circle to unify any emotional state within it. On the proposed Emotion Circle, each emotion distribution is represented with an emotion vector, which is defined with three attributes.
  arXiv  Detail & Related papers  (2021-06-23T14:53:27Z)
• Enhancing Cognitive Models of Emotions with Representation Learning [58.2386408470585]
  We present a novel deep learning-based framework to generate embedding representations of fine-grained emotions. Our framework integrates a contextualized embedding encoder with a multi-head probing model. Our model is evaluated on the Empathetic Dialogue dataset and shows the state-of-the-art result for classifying 32 emotions.
  arXiv  Detail & Related papers  (2021-04-20T16:55:15Z)
• Emotion pattern detection on facial videos using functional statistics [62.997667081978825]
  We propose a technique based on Functional ANOVA to extract significant patterns of face muscles movements. We determine if there are time-related differences on expressions among emotional groups by using a functional F-test.
  arXiv  Detail & Related papers  (2021-03-01T08:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.