A Multibias-mitigated and Sentiment Knowledge Enriched Transformer for Debiasing in Multimodal Conversational Emotion Recognition

• URL: http://arxiv.org/abs/2207.08104v1
• Date: Sun, 17 Jul 2022 08:16:49 GMT
• Title: A Multibias-mitigated and Sentiment Knowledge Enriched Transformer for Debiasing in Multimodal Conversational Emotion Recognition
• Authors: Jinglin Wang, Fang Ma, Yazhou Zhang, Dawei Song
• Abstract summary: Multimodal emotion recognition in conversations (mERC) is an active research topic in natural language processing (NLP) Innumerable implicit prejudices and preconceptions fill human language and conversations. Existing data-driven mERC approaches may offer higher emotional scores on utterances by females than males.
• Score: 9.020664590692705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal emotion recognition in conversations (mERC) is an active research topic in natural language processing (NLP), which aims to predict human's emotional states in communications of multiple modalities, e,g., natural language and facial gestures. Innumerable implicit prejudices and preconceptions fill human language and conversations, leading to the question of whether the current data-driven mERC approaches produce a biased error. For example, such approaches may offer higher emotional scores on the utterances by females than males. In addition, the existing debias models mainly focus on gender or race, where multibias mitigation is still an unexplored task in mERC. In this work, we take the first step to solve these issues by proposing a series of approaches to mitigate five typical kinds of bias in textual utterances (i.e., gender, age, race, religion and LGBTQ+) and visual representations (i.e, gender and age), followed by a Multibias-Mitigated and sentiment Knowledge Enriched bi-modal Transformer (MMKET). Comprehensive experimental results show the effectiveness of the proposed model and prove that the debias operation has a great impact on the classification performance for mERC. We hope our study will benefit the development of bias mitigation in mERC and related emotion studies.

Related papers

• Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs) [82.57490175399693]
  We study gender bias in 22 popular image-to-text vision-language assistants (VLAs) Our results show that VLAs replicate human biases likely present in the data, such as real-world occupational imbalances. To eliminate the gender bias in these models, we find that finetuning-based debiasing methods achieve the best tradeoff between debiasing and retaining performance on downstream tasks.
  arXiv  Detail & Related papers  (2024-10-25T05:59:44Z)
• Spoken Stereoset: On Evaluating Social Bias Toward Speaker in Speech Large Language Models [50.40276881893513]
  This study introduces Spoken Stereoset, a dataset specifically designed to evaluate social biases in Speech Large Language Models (SLLMs) By examining how different models respond to speech from diverse demographic groups, we aim to identify these biases. The findings indicate that while most models show minimal bias, some still exhibit slightly stereotypical or anti-stereotypical tendencies.
  arXiv  Detail & Related papers  (2024-08-14T16:55:06Z)
• Locating and Mitigating Gender Bias in Large Language Models [40.78150878350479]
  Large language models (LLM) are pre-trained on extensive corpora to learn facts and human cognition which contain human preferences. This process can inadvertently lead to these models acquiring biases and prevalent stereotypes in society. We propose the LSDM (Least Square Debias Method), a knowledge-editing based method for mitigating gender bias in occupational pronouns.
  arXiv  Detail & Related papers  (2024-03-21T13:57:43Z)
• Towards Multimodal Sentiment Analysis Debiasing via Bias Purification [21.170000473208372]
  Multimodal Sentiment Analysis (MSA) aims to understand human intentions by integrating emotion-related clues from diverse modalities. MSA task invariably suffers from unplanned dataset biases, particularly multimodal utterance-level label bias and word-level context bias. We present a Multimodal Counterfactual Inference Sentiment analysis framework based on causality rather than conventional likelihood.
  arXiv  Detail & Related papers  (2024-03-08T03:55:27Z)
• SEGAA: A Unified Approach to Predicting Age, Gender, and Emotion in Speech [0.0]
  This study ventures into predicting age, gender, and emotion from vocal cues, a field with vast applications. Exploring deep learning models for these predictions involves comparing single, multi-output, and sequential models highlighted in this paper. The experiments suggest that Multi-output models perform comparably to individual models, efficiently capturing the intricate relationships between variables and speech inputs, all while achieving improved runtime.
  arXiv  Detail & Related papers  (2024-03-01T11:28:37Z)
• Dynamic Causal Disentanglement Model for Dialogue Emotion Detection [77.96255121683011]
  We propose a Dynamic Causal Disentanglement Model based on hidden variable separation. This model effectively decomposes the content of dialogues and investigates the temporal accumulation of emotions. Specifically, we propose a dynamic temporal disentanglement model to infer the propagation of utterances and hidden variables.
  arXiv  Detail & Related papers  (2023-09-13T12:58:09Z)
• Blacks is to Anger as Whites is to Joy? Understanding Latent Affective Bias in Large Pre-trained Neural Language Models [3.5278693565908137]
  "Affective Bias" is biased association of emotions towards a particular gender, race, and religion. We show the existence of statistically significant affective bias in the PLM based emotion detection systems.
  arXiv  Detail & Related papers  (2023-01-21T20:23:09Z)
• Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Loss [80.79641247882012]
  We focus on unsupervised feature learning for Multimodal Emotion Recognition (MER) We consider discrete emotions, and as modalities text, audio and vision are used. Our method, as being based on contrastive loss between pairwise modalities, is the first attempt in MER literature.
  arXiv  Detail & Related papers  (2022-07-23T10:11:24Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)
• Variants of BERT, Random Forests and SVM approach for Multimodal Emotion-Target Sub-challenge [11.71437054341057]
  We present and discuss our classification methodology for MuSe-Topic Sub-challenge. We ensemble two language models which are ALBERT and RoBERTa to predict 10 classes of topics.
  arXiv  Detail & Related papers  (2020-07-28T01:15:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.