An Experimental Study of The Effects of Position Bias on Emotion CauseExtraction

• URL: http://arxiv.org/abs/2007.15066v1
• Date: Thu, 16 Jul 2020 08:02:36 GMT
• Title: An Experimental Study of The Effects of Position Bias on Emotion CauseExtraction
• Authors: Jiayuan Ding, Mayank Kejriwal
• Abstract summary: We show that a simple random selection approach toward Emotion Cause Extraction achieves similar performance compared to the baselines. An imbalance of emotional cause location exists in the benchmark, with a majority of cause clauses immediately preceding the central emotion clause. We conclude that it is the innate bias in this benchmark that caused high accuracy rate of these deep learning models in ECE.
• Score: 8.43954669406248
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Emotion Cause Extraction (ECE) aims to identify emotion causes from a document after annotating the emotion keywords. Some baselines have been proposed to address this problem, such as rule-based, commonsense based and machine learning methods. We show, however, that a simple random selection approach toward ECE that does not require observing the text achieves similar performance compared to the baselines. We utilized only position information relative to the emotion cause to accomplish this goal. Since position information alone without observing the text resulted in higher F-measure, we therefore uncovered a bias in the ECE single genre Sina-news benchmark. Further analysis showed that an imbalance of emotional cause location exists in the benchmark, with a majority of cause clauses immediately preceding the central emotion clause. We examine the bias from a linguistic perspective, and show that high accuracy rate of current state-of-art deep learning models that utilize location information is only evident in datasets that contain such position biases. The accuracy drastically reduced when a dataset with balanced location distribution is introduced. We therefore conclude that it is the innate bias in this benchmark that caused high accuracy rate of these deep learning models in ECE. We hope that the case study in this paper presents both a cautionary lesson, as well as a template for further studies, in interpreting the superior fit of deep learning models without checking for bias.

Related papers

• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs) Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings. By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Semantic Properties of cosine based bias scores for word embeddings [48.0753688775574]
  We propose requirements for bias scores to be considered meaningful for quantifying biases. We analyze cosine based scores from the literature with regard to these requirements. We underline these findings with experiments to show that the bias scores' limitations have an impact in the application case.
  arXiv  Detail & Related papers  (2024-01-27T20:31:10Z)
• Current Topological and Machine Learning Applications for Bias Detection in Text [4.799066966918178]
  This study utilizes the RedditBias database to analyze textual biases. Four transformer models, including BERT and RoBERTa variants, were explored. Findings suggest BERT, particularly mini BERT, excels in bias classification, while multilingual models lag.
  arXiv  Detail & Related papers  (2023-11-22T16:12:42Z)
• Dissecting Causal Biases [0.0]
  This paper focuses on a class of bias originating in the way training data is generated and/or collected. Four sources of bias are considered, namely, confounding, selection, measurement, and interaction.
  arXiv  Detail & Related papers  (2023-10-20T09:12:10Z)
• Towards Debiasing Frame Length Bias in Text-Video Retrieval via Causal Intervention [72.12974259966592]
  We present a unique and systematic study of a temporal bias due to frame length discrepancy between training and test sets of trimmed video clips. We propose a causal debiasing approach and perform extensive experiments and ablation studies on the Epic-Kitchens-100, YouCook2, and MSR-VTT datasets.
  arXiv  Detail & Related papers  (2023-09-17T15:58:27Z)
• Debiasing Stance Detection Models with Counterfactual Reasoning and Adversarial Bias Learning [15.68462203989933]
  Stance detection models tend to rely on dataset bias in the text part as a shortcut. We propose an adversarial bias learning module to model the bias more accurately.
  arXiv  Detail & Related papers  (2022-12-20T16:20:56Z)
• The SAME score: Improved cosine based bias score for word embeddings [49.75878234192369]
  We introduce SAME, a novel bias score for semantic bias in embeddings. We show that SAME is capable of measuring semantic bias and identify potential causes for social bias in downstream tasks.
  arXiv  Detail & Related papers  (2022-03-28T09:28:13Z)
• Understanding and Mitigating Annotation Bias in Facial Expression Recognition [3.325054486984015]
  Most existing works assume that human-generated annotations can be considered gold-standard and unbiased. We focus on facial expression recognition and compare the label biases between lab-controlled and in-the-wild datasets. We propose an AU-Calibrated Facial Expression Recognition framework that utilizes facial action units (AUs) and incorporates the triplet loss into the objective function.
  arXiv  Detail & Related papers  (2021-08-19T05:28:07Z)
• Position Bias Mitigation: A Knowledge-Aware Graph Model for Emotion Cause Extraction [24.288475819004034]
  We observe that a widely-used ECE dataset exhibits a bias that the majority of annotated cause clauses are either directly before their associated emotion clauses or are the emotion clauses themselves. Existing models for ECE tend to explore such relative position information and suffer from the dataset bias. We propose a novel strategy to generate adversarial examples in which the relative position information is no longer the indicative feature of cause clauses.
  arXiv  Detail & Related papers  (2021-06-07T11:14:58Z)
• LOGAN: Local Group Bias Detection by Clustering [86.38331353310114]
  We argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. We propose LOGAN, a new bias detection technique based on clustering. Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region.
  arXiv  Detail & Related papers  (2020-10-06T16:42:51Z)
• REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets [64.76453161039973]
  REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset. It surfacing potential biases along three dimensions: (1) object-based, (2) person-based, and (3) geography-based.
  arXiv  Detail & Related papers  (2020-04-16T23:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.