Subtle Misogyny Detection and Mitigation: An Expert-Annotated Dataset

• URL: http://arxiv.org/abs/2311.09443v1
• Date: Wed, 15 Nov 2023 23:27:19 GMT
• Title: Subtle Misogyny Detection and Mitigation: An Expert-Annotated Dataset
• Authors: Brooklyn Sheppard, Anna Richter, Allison Cohen, Elizabeth Allyn Smith, Tamara Kneese, Carolyne Pelletier, Ioana Baldini, Yue Dong
• Abstract summary: The Biasly dataset is built in collaboration with multi-disciplinary experts and annotators themselves. The dataset can be used for a range of NLP tasks, including classification, severity score regression, and text generation for rewrites.
• Score: 5.528106559459623
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Using novel approaches to dataset development, the Biasly dataset captures the nuance and subtlety of misogyny in ways that are unique within the literature. Built in collaboration with multi-disciplinary experts and annotators themselves, the dataset contains annotations of movie subtitles, capturing colloquial expressions of misogyny in North American film. The dataset can be used for a range of NLP tasks, including classification, severity score regression, and text generation for rewrites. In this paper, we discuss the methodology used, analyze the annotations obtained, and provide baselines using common NLP algorithms in the context of misogyny detection and mitigation. We hope this work will promote AI for social good in NLP for bias detection, explanation, and removal.

Related papers

• Mitigating Biases to Embrace Diversity: A Comprehensive Annotation Benchmark for Toxic Language [0.0]
  This study introduces a prescriptive annotation benchmark grounded in humanities research to ensure consistent, unbiased labeling of offensive language. We contribute two newly annotated datasets that achieve higher inter-annotator agreement between human and language model (LLM) annotations.
  arXiv  Detail & Related papers  (2024-10-17T08:10:24Z)
• Beyond Coarse-Grained Matching in Video-Text Retrieval [50.799697216533914]
  We introduce a new approach for fine-grained evaluation. Our approach can be applied to existing datasets by automatically generating hard negative test captions. Experiments on our fine-grained evaluations demonstrate that this approach enhances a model's ability to understand fine-grained differences.
  arXiv  Detail & Related papers  (2024-10-16T09:42:29Z)
• Paired Completion: Flexible Quantification of Issue-framing at Scale with LLMs [0.41436032949434404]
  We develop and rigorously evaluate new detection methods for issue framing and narrative analysis within large text datasets. We show that issue framing can be reliably and efficiently detected in large corpora with only a few examples of either perspective on a given issue.
  arXiv  Detail & Related papers  (2024-08-19T07:14:15Z)
• MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types [68.76742370525234]
  We propose a new evaluation scheme to model human judgments in 7 NLP tasks, based on the fine-grained mismatches between a pair of texts. Inspired by the recent efforts in several NLP tasks for fine-grained evaluation, we introduce a set of 13 mismatch error types. We show that the mismatch errors between the sentence pairs on the held-out datasets from 7 NLP tasks align well with the human evaluation.
  arXiv  Detail & Related papers  (2023-06-18T01:38:53Z)
• Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future [63.99570204416711]
  We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets. We define a uniform evaluation setup including a new formalization of the annotation error detection task. We release our datasets and implementations in an easy-to-use and open source software package.
  arXiv  Detail & Related papers  (2022-06-05T22:31:45Z)
• Detecting Textual Adversarial Examples Based on Distributional Characteristics of Data Representations [11.93653349589025]
  adversarial examples are constructed by adding small non-random perturbations to correctly classified inputs. Approaches to adversarial attacks in natural language tasks have boomed in the last five years using character-level, word-level, or phrase-level perturbations. We propose two new reactive methods for NLP to fill this gap. Adapted LID and MDRE obtain state-of-the-art results on character-level, word-level, and phrase-level attacks on the IMDB dataset.
  arXiv  Detail & Related papers  (2022-04-29T02:32:02Z)
• Word Embeddings via Causal Inference: Gender Bias Reducing and Semantic Information Preserving [3.114945725130788]
  We propose a novel methodology that leverages a causal inference framework to effectively remove gender bias. Our comprehensive experiments show that the proposed method achieves state-of-the-art results in gender-debiasing tasks.
  arXiv  Detail & Related papers  (2021-12-09T19:57:22Z)
• A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation [50.55448707570669]
  We propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes. To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations.
  arXiv  Detail & Related papers  (2021-04-18T04:09:48Z)
• The Gap on GAP: Tackling the Problem of Differing Data Distributions in Bias-Measuring Datasets [58.53269361115974]
  Diagnostic datasets that can detect biased models are an important prerequisite for bias reduction within natural language processing. undesired patterns in the collected data can make such tests incorrect. We introduce a theoretically grounded method for weighting test samples to cope with such patterns in the test data.
  arXiv  Detail & Related papers  (2020-11-03T16:50:13Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Improving unsupervised neural aspect extraction for online discussions using out-of-domain classification [11.746330029375745]
  We introduce a simple approach based on sentence filtering to improve topical aspects learned from newsgroups-based content. The positive effect of sentence filtering on topic coherence is demonstrated in comparison to aspect extraction models trained on unfiltered texts.
  arXiv  Detail & Related papers  (2020-06-17T10:34:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.