It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection

• URL: http://arxiv.org/abs/2204.07660v1
• Date: Fri, 15 Apr 2022 22:08:45 GMT
• Title: It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection
• Authors: Youssef Mohamed, Faizan Farooq Khan, Kilichbek Haydarov, Mohamed Elhoseiny
• Abstract summary: ArtEmis was recently introduced as a large-scale dataset of emotional reactions to images along with language explanations. We observed a significant emotional bias towards instance-rich emotions, making trained neural speakers less accurate in describing under-represented emotions. We propose a contrastive data collection approach to balance ArtEmis with a new complementary dataset.
• Score: 22.209744124318966
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Datasets that capture the connection between vision, language, and affection are limited, causing a lack of understanding of the emotional aspect of human intelligence. As a step in this direction, the ArtEmis dataset was recently introduced as a large-scale dataset of emotional reactions to images along with language explanations of these chosen emotions. We observed a significant emotional bias towards instance-rich emotions, making trained neural speakers less accurate in describing under-represented emotions. We show that collecting new data, in the same way, is not effective in mitigating this emotional bias. To remedy this problem, we propose a contrastive data collection approach to balance ArtEmis with a new complementary dataset such that a pair of similar images have contrasting emotions (one positive and one negative). We collected 260,533 instances using the proposed method, we combine them with ArtEmis, creating a second iteration of the dataset. The new combined dataset, dubbed ArtEmis v2.0, has a balanced distribution of emotions with explanations revealing more fine details in the associated painting. Our experiments show that neural speakers trained on the new dataset improve CIDEr and METEOR evaluation metrics by 20% and 7%, respectively, compared to the biased dataset. Finally, we also show that the performance per emotion of neural speakers is improved across all the emotion categories, significantly on under-represented emotions. The collected dataset and code are available at https://artemisdataset-v2.org.

Related papers

• EMO-KNOW: A Large Scale Dataset on Emotion and Emotion-cause [8.616061735005314]
  We introduce a large-scale dataset of emotion causes, derived from 9.8 million cleaned tweets over 15 years. The novelty of our dataset stems from its broad spectrum of emotion classes and the abstractive emotion cause. Our dataset will enable the design of emotion-aware systems that account for the diverse emotional responses of different people.
  arXiv  Detail & Related papers  (2024-06-18T08:26:33Z)
• Language Models (Mostly) Do Not Consider Emotion Triggers When Predicting Emotion [87.18073195745914]
  We investigate how well human-annotated emotion triggers correlate with features deemed salient in their prediction of emotions. Using EmoTrigger, we evaluate the ability of large language models to identify emotion triggers. Our analysis reveals that emotion triggers are largely not considered salient features for emotion prediction models, instead there is intricate interplay between various features and the task of emotion detection.
  arXiv  Detail & Related papers  (2023-11-16T06:20:13Z)
• Data Augmentation for Emotion Detection in Small Imbalanced Text Data [0.0]
  One of the challenges is the shortage of available datasets that have been annotated with emotions. We studied the impact of data augmentation techniques precisely when applied to small imbalanced datasets. Our experimental results show that using the augmented data when training the classifier model leads to significant improvements.
  arXiv  Detail & Related papers  (2023-10-25T21:29:36Z)
• EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes [53.95428298229396]
  We introduce EmoSet, the first large-scale visual emotion dataset annotated with rich attributes. EmoSet comprises 3.3 million images in total, with 118,102 of these images carefully labeled by human annotators. Motivated by psychological studies, in addition to emotion category, each image is also annotated with a set of describable emotion attributes.
  arXiv  Detail & Related papers  (2023-07-16T06:42:46Z)
• Reevaluating Data Partitioning for Emotion Detection in EmoWOZ [0.0]
  EmoWoz is an extension of MultiWOZ that provides emotion labels for the dialogues. MultiWOZ was partitioned initially for another purpose, resulting in a distributional shift when considering the new purpose of emotion recognition. We propose a stratified sampling scheme based on emotion tags to address this issue, improve the dataset's distribution, and reduce dataset shift.
  arXiv  Detail & Related papers  (2023-03-15T03:06:13Z)
• A cross-corpus study on speech emotion recognition [29.582678406878568]
  This study investigates whether information learnt from acted emotions is useful for detecting natural emotions. Four adult English datasets covering acted, elicited and natural emotions are considered. A state-of-the-art model is proposed to accurately investigate the degradation of performance.
  arXiv  Detail & Related papers  (2022-07-05T15:15:22Z)
• SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network [83.27291945217424]
  We propose a novel Scene-Object interreLated Visual Emotion Reasoning network (SOLVER) to predict emotions from images. To mine the emotional relationships between distinct objects, we first build up an Emotion Graph based on semantic concepts and visual features. We also design a Scene-Object Fusion Module to integrate scenes and objects, which exploits scene features to guide the fusion process of object features with the proposed scene-based attention mechanism.
  arXiv  Detail & Related papers  (2021-10-24T02:41:41Z)
• Affective Image Content Analysis: Two Decades Review and New Perspectives [132.889649256384]
  We will comprehensively review the development of affective image content analysis (AICA) in the recent two decades. We will focus on the state-of-the-art methods with respect to three main challenges -- the affective gap, perception subjectivity, and label noise and absence. We discuss some challenges and promising research directions in the future, such as image content and context understanding, group emotion clustering, and viewer-image interaction.
  arXiv  Detail & Related papers  (2021-06-30T15:20:56Z)
• A Circular-Structured Representation for Visual Emotion Distribution Learning [82.89776298753661]
  We propose a well-grounded circular-structured representation to utilize the prior knowledge for visual emotion distribution learning. To be specific, we first construct an Emotion Circle to unify any emotional state within it. On the proposed Emotion Circle, each emotion distribution is represented with an emotion vector, which is defined with three attributes.
  arXiv  Detail & Related papers  (2021-06-23T14:53:27Z)
• Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition [55.44502358463217]
  We propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. Our model achieves state-of-the-art performance on most of the emotion categories. Our model also outperforms existing baselines in the zero-shot and few-shot scenarios for unseen emotions.
  arXiv  Detail & Related papers  (2020-09-21T06:10:39Z)
• Context Based Emotion Recognition using EMOTIC Dataset [22.631542327834595]
  We present EMOTIC, a dataset of images of people annotated with their apparent emotion. Using the EMOTIC dataset we train different CNN models for emotion recognition. Our results show how scene context provides important information to automatically recognize emotional states.
  arXiv  Detail & Related papers  (2020-03-30T12:38:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.