Measuring a Texts Fairness Dimensions Using Machine Learning Based on Social Psychological Factors

• URL: http://arxiv.org/abs/2111.00086v1
• Date: Fri, 29 Oct 2021 21:09:17 GMT
• Title: Measuring a Texts Fairness Dimensions Using Machine Learning Based on Social Psychological Factors
• Abstract summary: We use psychology to determine the principal factors that humans use when making a fairness assessment. We digitise these using word embeddings into a multi-dimensioned sentence level fairness perceptions vector. A second approach, using PCA and ML based on the said fairness approximation vector produces an F1 score of 86.2.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fairness is a principal social value that can be observed in civilisations around the world. A manifestations of this is in social agreements, often described in texts, such as contracts. Yet, despite the prevalence of such, a fairness metric for texts describing a social act remains wanting. To address this, we take a step back to consider the problem based on first principals. Instead of using rules or templates, we utilise social psychology literature to determine the principal factors that humans use when making a fairness assessment. We then attempt to digitise these using word embeddings into a multi-dimensioned sentence level fairness perceptions vector to serve as an approximation for these fairness perceptions. The method leverages a pro-social bias within word embeddings, for which we obtain an F1= 81.0. A second approach, using PCA and ML based on the said fairness approximation vector produces an F1 score of 86.2. We details improvements that can be made in the methodology to incorporate the projection of sentence embedding on to a subspace representation of fairness.

Related papers

• Subjective fairness in algorithmic decision-support [0.0]
  The treatment of fairness in decision-making literature usually involves quantifying fairness using objective measures. This work takes a critical stance to highlight the limitations of these approaches using sociological insights. We redefine fairness as a subjective property moving from a top-down to a bottom-up approach.
  arXiv  Detail & Related papers  (2024-06-28T14:37:39Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• COFFEE: Counterfactual Fairness for Personalized Text Generation in Explainable Recommendation [56.520470678876656]
  bias inherent in user written text can associate different levels of linguistic quality with users' protected attributes. We introduce a general framework to achieve measure-specific counterfactual fairness in explanation generation.
  arXiv  Detail & Related papers  (2022-10-14T02:29:10Z)
• A Novel Approach to Fairness in Automated Decision-Making using Affective Normalization [2.0178765779788495]
  We propose a method for measuring the affective, socially biased, component, thus enabling its removal. That is, given a decision-making process, these affective measurements remove the affective bias in the decision, rendering it fair across a set of categories defined by the method itself.
  arXiv  Detail & Related papers  (2022-05-02T11:48:53Z)
• Measuring Fairness of Text Classifiers via Prediction Sensitivity [63.56554964580627]
  ACCUMULATED PREDICTION SENSITIVITY measures fairness in machine learning models based on the model's prediction sensitivity to perturbations in input features. We show that the metric can be theoretically linked with a specific notion of group fairness (statistical parity) and individual fairness.
  arXiv  Detail & Related papers  (2022-03-16T15:00:33Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)
• Fairness for Whom? Understanding the Reader's Perception of Fairness in Text Summarization [9.136419921943235]
  We study the interplay between the fairness notions and how readers perceive them in textual summaries. Standard ROUGE evaluation metrics are unable to quantify the perceived (un)fairness of the summaries.
  arXiv  Detail & Related papers  (2021-01-29T05:14:34Z)
• Towards Debiasing Sentence Representations [109.70181221796469]
  We show that Sent-Debias is effective in removing biases, and at the same time, preserves performance on sentence-level downstream tasks. We hope that our work will inspire future research on characterizing and removing social biases from widely adopted sentence representations for fairer NLP.
  arXiv  Detail & Related papers  (2020-07-16T04:22:30Z)
• Machine learning fairness notions: Bridging the gap with real-world applications [4.157415305926584]
  Fairness emerged as an important requirement to guarantee that Machine Learning predictive systems do not discriminate against specific individuals or entire sub-populations. This paper is a survey that illustrates the subtleties between fairness notions through a large number of examples and scenarios.
  arXiv  Detail & Related papers  (2020-06-30T13:01:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.