Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of
Movie Dialogues
- URL: http://arxiv.org/abs/2205.15951v2
- Date: Wed, 1 Jun 2022 05:43:53 GMT
- Title: Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of
Movie Dialogues
- Authors: Sandhya Singh, Prapti Roy, Nihar Sahoo, Niteesh Mallela, Himanshu
Gupta, Pushpak Bhattacharyya, Milind Savagaonkar, Nidhi, Roshni Ramnani,
Anutosh Maitra, Shubhashis Sengupta
- Abstract summary: Social biases and stereotypes present in movies can cause extensive damage due to their reach.
We introduce a new dataset of movie scripts that are annotated for identity bias.
The dataset contains dialogue turns annotated for (i) bias labels for seven categories, viz., gender, race/ethnicity, religion, age, occupation, LGBTQ, and other.
- Score: 20.222820874864748
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Movies reflect society and also hold power to transform opinions. Social
biases and stereotypes present in movies can cause extensive damage due to
their reach. These biases are not always found to be the need of storyline but
can creep in as the author's bias. Movie production houses would prefer to
ascertain that the bias present in a script is the story's demand. Today, when
deep learning models can give human-level accuracy in multiple tasks, having an
AI solution to identify the biases present in the script at the writing stage
can help them avoid the inconvenience of stalled release, lawsuits, etc. Since
AI solutions are data intensive and there exists no domain specific data to
address the problem of biases in scripts, we introduce a new dataset of movie
scripts that are annotated for identity bias. The dataset contains dialogue
turns annotated for (i) bias labels for seven categories, viz., gender,
race/ethnicity, religion, age, occupation, LGBTQ, and other, which contains
biases like body shaming, personality bias, etc. (ii) labels for sensitivity,
stereotype, sentiment, emotion, emotion intensity, (iii) all labels annotated
with context awareness, (iv) target groups and reason for bias labels and (v)
expert-driven group-validation process for high quality annotations. We also
report various baseline performances for bias identification and category
detection on our dataset.
Related papers
- White Men Lead, Black Women Help? Benchmarking Language Agency Social Biases in LLMs [58.27353205269664]
Language agency is an important aspect of evaluating social biases in texts.
Previous research often relies on string-matching techniques to identify agentic and communal words.
We introduce the novel Language Agency Bias Evaluation benchmark.
arXiv Detail & Related papers (2024-04-16T12:27:54Z) - ''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT
Generated English Text [11.085070600065801]
Language serves as a powerful tool for the manifestation of societal belief systems.
Gender bias is one of the most pervasive biases in our society.
We create the first dataset of GPT-generated English text with normative ratings of gender bias.
arXiv Detail & Related papers (2023-10-26T14:34:06Z) - CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI
Collaboration for Large Language Models [52.25049362267279]
We present a Chinese Bias Benchmark dataset that consists of over 100K questions jointly constructed by human experts and generative language models.
The testing instances in the dataset are automatically derived from 3K+ high-quality templates manually authored with stringent quality control.
Extensive experiments demonstrate the effectiveness of the dataset in detecting model bias, with all 10 publicly available Chinese large language models exhibiting strong bias in certain categories.
arXiv Detail & Related papers (2023-06-28T14:14:44Z) - VisoGender: A dataset for benchmarking gender bias in image-text pronoun
resolution [80.57383975987676]
VisoGender is a novel dataset for benchmarking gender bias in vision-language models.
We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas.
We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes.
arXiv Detail & Related papers (2023-06-21T17:59:51Z) - Detecting Unintended Social Bias in Toxic Language Datasets [32.724030288421474]
This paper introduces a new dataset ToxicBias curated from the existing dataset of Kaggle competition named "Jigsaw Unintended Bias in Toxicity Classification"
The dataset contains instances annotated for five different bias categories, viz., gender, race/ethnicity, religion, political, and LGBTQ.
We train transformer-based models using our curated datasets and report baseline performance for bias identification, target generation, and bias implications.
arXiv Detail & Related papers (2022-10-21T06:50:12Z) - Neural Media Bias Detection Using Distant Supervision With BABE -- Bias
Annotations By Experts [24.51774048437496]
This paper presents BABE, a robust and diverse data set for media bias research.
It consists of 3,700 sentences balanced among topics and outlets, containing media bias labels on the word and sentence level.
Based on our data, we also introduce a way to detect bias-inducing sentences in news articles automatically.
arXiv Detail & Related papers (2022-09-29T05:32:55Z) - The SAME score: Improved cosine based bias score for word embeddings [63.24247894974291]
We provide a bias definition based on the ideas from the literature and derive novel requirements for bias scores.
We propose a new bias score, SAME, to address the shortcomings of existing bias scores and show empirically that SAME is better suited to quantify biases in word embeddings.
arXiv Detail & Related papers (2022-03-28T09:28:13Z) - Uncovering Implicit Gender Bias in Narratives through Commonsense
Inference [21.18458377708873]
We study gender biases associated with the protagonist in model-generated stories.
We focus on implicit biases, and use a commonsense reasoning engine to uncover them.
arXiv Detail & Related papers (2021-09-14T04:57:45Z) - PowerTransformer: Unsupervised Controllable Revision for Biased Language
Correction [62.46299488465803]
We formulate a new revision task that aims to rewrite a given text to correct the implicit and potentially undesirable bias in character portrayals.
We introduce PowerTransformer as an approach that debiases text through the lens of connotation frames.
We demonstrate that our approach outperforms ablations and existing methods from related tasks.
arXiv Detail & Related papers (2020-10-26T18:05:48Z) - Multi-Dimensional Gender Bias Classification [67.65551687580552]
Machine learning models can inadvertently learn socially undesirable patterns when training on gender biased text.
We propose a general framework that decomposes gender bias in text along several pragmatic and semantic dimensions.
Using this fine-grained framework, we automatically annotate eight large scale datasets with gender information.
arXiv Detail & Related papers (2020-05-01T21:23:20Z) - Detection and Mitigation of Bias in Ted Talk Ratings [3.3598755777055374]
Implicit bias is a behavioral conditioning that leads us to attribute predetermined characteristics to members of certain groups.
This paper quantifies implicit bias in viewer ratings of TEDTalks, a diverse social platform assessing social and professional performance.
arXiv Detail & Related papers (2020-03-02T06:13:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.