Controlled Analyses of Social Biases in Wikipedia Bios

• URL: http://arxiv.org/abs/2101.00078v1
• Date: Thu, 31 Dec 2020 21:27:12 GMT
• Title: Controlled Analyses of Social Biases in Wikipedia Bios
• Authors: Anjalie Field, Chan Young Park, Yulia Tsvetkov
• Abstract summary: We present a methodology for reducing the effects of confounding variables in analyses of Wikipedia biography pages. We evaluate our methodology by developing metrics to measure how well the comparison corpus aligns with the target corpus. Our results show that failing to control for confounding variables can result in different conclusions and mask biases.
• Score: 27.591896251854724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Social biases on Wikipedia, a widely-read global platform, could greatly influence public opinion. While prior research has examined man/woman gender bias in biography articles, possible influences of confounding variables limit conclusions. In this work, we present a methodology for reducing the effects of confounding variables in analyses of Wikipedia biography pages. Given a target corpus for analysis (e.g. biography pages about women), we present a method for constructing a comparison corpus that matches the target corpus in as many attributes as possible, except the target attribute (e.g. the gender of the subject). We evaluate our methodology by developing metrics to measure how well the comparison corpus aligns with the target corpus. We then examine how articles about gender and racial minorities (cisgender women, non-binary people, transgender women, and transgender men; African American, Asian American, and Hispanic/Latinx American people) differ from other articles, including analyses driven by social theories like intersectionality. In addition to identifying suspect social biases, our results show that failing to control for confounding variables can result in different conclusions and mask biases. Our contributions include methodology that facilitates further analyses of bias in Wikipedia articles, findings that can aid Wikipedia editors in reducing biases, and framework and evaluation metrics to guide future work in this area.

Related papers

• The Impact of Debiasing on the Performance of Language Models in Downstream Tasks is Underestimated [70.23064111640132]
  We compare the impact of debiasing on performance across multiple downstream tasks using a wide-range of benchmark datasets. Experiments show that the effects of debiasing are consistently emphunderestimated across all tasks.
  arXiv  Detail & Related papers  (2023-09-16T20:25:34Z)
• Wikibio: a Semantic Resource for the Intersectional Analysis of Biographical Events [3.8455936323976694]
  We present a new corpus annotated for biographical event detection. The model was able to detect all mentions of the target-entity in a biography with an F-score of 0.808. It was also used for performing an analysis of biases about women and non-Western people in Wikipedia biographies.
  arXiv  Detail & Related papers  (2023-06-15T20:59:37Z)
• Gender Biases in Automatic Evaluation Metrics for Image Captioning [87.15170977240643]
  We conduct a systematic study of gender biases in model-based evaluation metrics for image captioning tasks. We demonstrate the negative consequences of using these biased metrics, including the inability to differentiate between biased and unbiased generations. We present a simple and effective way to mitigate the metric bias without hurting the correlations with human judgments.
  arXiv  Detail & Related papers  (2023-05-24T04:27:40Z)
• Choose Your Lenses: Flaws in Gender Bias Evaluation [29.16221451643288]
  We assess the current paradigm of gender bias evaluation and identify several flaws in it. First, we highlight the importance of extrinsic bias metrics that measure how a model's performance on some task is affected by gender. Second, we find that datasets and metrics are often coupled, and discuss how their coupling hinders the ability to obtain reliable conclusions.
  arXiv  Detail & Related papers  (2022-10-20T17:59:55Z)
• Social Biases in Automatic Evaluation Metrics for NLG [53.76118154594404]
  We propose an evaluation method based on Word Embeddings Association Test (WEAT) and Sentence Embeddings Association Test (SEAT) to quantify social biases in evaluation metrics. We construct gender-swapped meta-evaluation datasets to explore the potential impact of gender bias in image caption and text summarization tasks.
  arXiv  Detail & Related papers  (2022-10-17T08:55:26Z)
• Theories of "Gender" in NLP Bias Research [0.0]
  We survey nearly 200 articles concerning gender bias in NLP. We find that the majority of the articles do not make their theorization of gender explicit. Many conflate sex characteristics, social gender, and linguistic gender in ways that disregard the existence and experience of trans, nonbinary, and intersex people.
  arXiv  Detail & Related papers  (2022-05-05T09:20:53Z)
• Anatomizing Bias in Facial Analysis [86.79402670904338]
  Existing facial analysis systems have been shown to yield biased results against certain demographic subgroups. It has become imperative to ensure that these systems do not discriminate based on gender, identity, or skin tone of individuals. This has led to research in the identification and mitigation of bias in AI systems.
  arXiv  Detail & Related papers  (2021-12-13T09:51:13Z)
• Men Are Elected, Women Are Married: Events Gender Bias on Wikipedia [16.892630505629224]
  We present the first event-centric study of gender biases in a Wikipedia corpus. We detect events with a state-of-the-art event detection model, calibrate the results using strategically generated templates, and extract events that have asymmetric associations with genders.
  arXiv  Detail & Related papers  (2021-06-03T05:22:16Z)
• Gender bias in magazines oriented to men and women: a computational approach [58.720142291102135]
  We compare the content of a women-oriented magazine with that of a men-oriented one, both produced by the same editorial group over a decade. With Topic Modelling techniques we identify the main themes discussed in the magazines and quantify how much the presence of these topics differs between magazines over time. Our results show that the frequency of appearance of the topics Family, Business and Women as sex objects, present an initial bias that tends to disappear over time.
  arXiv  Detail & Related papers  (2020-11-24T14:02:49Z)
• Multilingual Contextual Affective Analysis of LGBT People Portrayals in Wikipedia [34.183132688084534]
  Specific lexical choices in narrative text reflect both the writer's attitudes towards people in the narrative and influence the audience's reactions. We show how word connotations differ across languages and cultures, highlighting the difficulty of generalizing existing English datasets. We then demonstrate the usefulness of our method by analyzing Wikipedia biography pages of members of the LGBT community across three languages.
  arXiv  Detail & Related papers  (2020-10-21T08:27:36Z)
• Gender Stereotype Reinforcement: Measuring the Gender Bias Conveyed by Ranking Algorithms [68.85295025020942]
  We propose the Gender Stereotype Reinforcement (GSR) measure, which quantifies the tendency of a Search Engines to support gender stereotypes. GSR is the first specifically tailored measure for Information Retrieval, capable of quantifying representational harms.
  arXiv  Detail & Related papers  (2020-09-02T20:45:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.