Subtle Misogyny Detection and Mitigation: An Expert-Annotated Dataset
- URL: http://arxiv.org/abs/2311.09443v1
- Date: Wed, 15 Nov 2023 23:27:19 GMT
- Title: Subtle Misogyny Detection and Mitigation: An Expert-Annotated Dataset
- Authors: Brooklyn Sheppard, Anna Richter, Allison Cohen, Elizabeth Allyn Smith,
Tamara Kneese, Carolyne Pelletier, Ioana Baldini, Yue Dong
- Abstract summary: The Biasly dataset is built in collaboration with multi-disciplinary experts and annotators themselves.
The dataset can be used for a range of NLP tasks, including classification, severity score regression, and text generation for rewrites.
- Score: 5.528106559459623
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Using novel approaches to dataset development, the Biasly dataset captures
the nuance and subtlety of misogyny in ways that are unique within the
literature. Built in collaboration with multi-disciplinary experts and
annotators themselves, the dataset contains annotations of movie subtitles,
capturing colloquial expressions of misogyny in North American film. The
dataset can be used for a range of NLP tasks, including classification,
severity score regression, and text generation for rewrites. In this paper, we
discuss the methodology used, analyze the annotations obtained, and provide
baselines using common NLP algorithms in the context of misogyny detection and
mitigation. We hope this work will promote AI for social good in NLP for bias
detection, explanation, and removal.
Related papers
- MISMATCH: Fine-grained Evaluation of Machine-generated Text with
Mismatch Error Types [68.76742370525234]
We propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts.
Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types.
We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.
arXiv Detail & Related papers (2023-06-18T01:38:53Z) - Annotation Error Detection: Analyzing the Past and Present for a More
Coherent Future [63.99570204416711]
We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets.
We define a uniform evaluation setup including a new formalization of the annotation error detection task.
We release our datasets and implementations in an easy-to-use and open source software package.
arXiv Detail & Related papers (2022-06-05T22:31:45Z) - Detecting Textual Adversarial Examples Based on Distributional
Characteristics of Data Representations [11.93653349589025]
adversarial examples are constructed by adding small non-random perturbations to correctly classified inputs.
Approaches to adversarial attacks in natural language tasks have boomed in the last five years using character-level, word-level, or phrase-level perturbations.
We propose two new reactive methods for NLP to fill this gap.
Adapted LID and MDRE obtain state-of-the-art results on character-level, word-level, and phrase-level attacks on the IMDB dataset.
arXiv Detail & Related papers (2022-04-29T02:32:02Z) - Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic
Information Preserving [3.114945725130788]
We propose a novel methodology that leverages a causal inference framework to effectively remove gender bias.
Our comprehensive experiments show that the proposed method achieves state-of-the-art results in gender-debiasing tasks.
arXiv Detail & Related papers (2021-12-09T19:57:22Z) - Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
We propose methods to facilitate identification of training data artifacts.
We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data.
We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
arXiv Detail & Related papers (2021-07-01T09:26:13Z) - A Token-level Reference-free Hallucination Detection Benchmark for
Free-form Text Generation [50.55448707570669]
We propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes.
To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations.
arXiv Detail & Related papers (2021-04-18T04:09:48Z) - A survey on extremism analysis using Natural Language Processing [7.885207996427683]
This survey aims to review the contributions of NLP to the field of extremism research.
The content includes a description and comparison of the frequently used NLP techniques, how they were applied and the insights they provided.
future trends, challenges and directions derived from these highlights are suggested.
arXiv Detail & Related papers (2021-03-28T11:05:43Z) - Let-Mi: An Arabic Levantine Twitter Dataset for Misogynistic Language [0.0]
We introduce an Arabic Levantine Twitter dataset for Misogynistic language (LeT-Mi) to be the first benchmark dataset for Arabic misogyny.
Let-Mi was used as an evaluation dataset through binary/multi-/target classification tasks conducted by several state-of-the-art machine learning systems.
arXiv Detail & Related papers (2021-03-18T12:01:13Z) - The Gap on GAP: Tackling the Problem of Differing Data Distributions in
Bias-Measuring Datasets [58.53269361115974]
Diagnostic datasets that can detect biased models are an important prerequisite for bias reduction within natural language processing.
undesired patterns in the collected data can make such tests incorrect.
We introduce a theoretically grounded method for weighting test samples to cope with such patterns in the test data.
arXiv Detail & Related papers (2020-11-03T16:50:13Z) - Weakly-Supervised Aspect-Based Sentiment Analysis via Joint
Aspect-Sentiment Topic Embedding [71.2260967797055]
We propose a weakly-supervised approach for aspect-based sentiment analysis.
We learn sentiment, aspect> joint topic embeddings in the word embedding space.
We then use neural models to generalize the word-level discriminative information.
arXiv Detail & Related papers (2020-10-13T21:33:24Z) - Improving unsupervised neural aspect extraction for online discussions
using out-of-domain classification [11.746330029375745]
We introduce a simple approach based on sentence filtering to improve topical aspects learned from newsgroups-based content.
The positive effect of sentence filtering on topic coherence is demonstrated in comparison to aspect extraction models trained on unfiltered texts.
arXiv Detail & Related papers (2020-06-17T10:34:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.