Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages

• URL: http://arxiv.org/abs/2205.12676v3
• Date: Wed, 12 Apr 2023 14:49:46 GMT
• Title: Evaluating the Diversity, Equity and Inclusion of NLP Technology: A Case Study for Indian Languages
• Authors: Simran Khanuja, Sebastian Ruder, Partha Talukdar
• Abstract summary: In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world's languages. We propose an evaluation paradigm that assesses NLP technologies across all three dimensions.
• Score: 35.86100962711644
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In order for NLP technology to be widely applicable, fair, and useful, it needs to serve a diverse set of speakers across the world's languages, be equitable, i.e., not unduly biased towards any particular language, and be inclusive of all users, particularly in low-resource settings where compute constraints are common. In this paper, we propose an evaluation paradigm that assesses NLP technologies across all three dimensions. While diversity and inclusion have received attention in recent literature, equity is currently unexplored. We propose to address this gap using the Gini coefficient, a well-established metric used for estimating societal wealth inequality. Using our paradigm, we highlight the distressed state of current technologies for Indian (IN) languages (a linguistically large and diverse set, with a varied speaker population), across all three dimensions. To improve upon these metrics, we demonstrate the importance of region-specific choices in model building and dataset creation, and more importantly, propose a novel, generalisable approach to optimal resource allocation during fine-tuning. Finally, we discuss steps to mitigate these biases and encourage the community to employ multi-faceted evaluation when building linguistically diverse and equitable technologies.

Related papers

• A Capabilities Approach to Studying Bias and Harm in Language Technologies [4.135516576952934]
  We consider fairness, bias, and inclusion in Language Technologies through the lens of the Capabilities Approach. The Capabilities Approach centers on what people are capable of achieving, given their intersectional social, political, and economic contexts. We detail the Capabilities Approach, its relationship to multilingual and multicultural evaluation, and how the framework affords meaningful collaboration with community members in defining and measuring the harms of Language Technologies.
  arXiv  Detail & Related papers  (2024-11-06T22:46:13Z)
• Unintended Impacts of LLM Alignment on Global Representation [62.6579934112071]
  We show that developers align Large Language Models (LLMs) to user preferences through a variety of procedures, such as Reinforcement Learning From Human Feedback (RLHF) and Direct Preference Optimization (DPO) We explore how alignment impacts performance along three axes of global representation: English dialects, multilingualism, and opinions from and about countries worldwide. We conclude by discussing design decisions that led to these unintended impacts and recommendations for more equitable preference tuning.
  arXiv  Detail & Related papers  (2024-02-22T23:31:22Z)
• What Do Dialect Speakers Want? A Survey of Attitudes Towards Language Technology for German Dialects [60.8361859783634]
  We survey speakers of dialects and regional languages related to German. We find that respondents are especially in favour of potential NLP tools that work with dialectal input.
  arXiv  Detail & Related papers  (2024-02-19T09:15:28Z)
• Natural Language Processing for Dialects of a Language: A Survey [56.93337350526933]
  State-of-the-art natural language processing (NLP) models are trained on massive training corpora, and report a superlative performance on evaluation datasets. This survey delves into an important attribute of these datasets: the dialect of a language. Motivated by the performance degradation of NLP models for dialectic datasets and its implications for the equity of language technologies, we survey past research in NLP for dialects in terms of datasets, and approaches.
  arXiv  Detail & Related papers  (2024-01-11T03:04:38Z)
• Towards Bridging the Digital Language Divide [4.234367850767171]
  multilingual language processing systems often exhibit a hardwired, yet usually involuntary and hidden representational preference towards certain languages. We show that biased technology is often the result of research and development methodologies that do not do justice to the complexity of the languages being represented. We present a new initiative that aims at reducing linguistic bias through both technological design and methodology.
  arXiv  Detail & Related papers  (2023-07-25T10:53:20Z)
• IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases [53.054598423181844]
  We present a novel NER cascade approach comprising three steps. We empirically demonstrate the significance of external knowledge bases in accurately classifying fine-grained and emerging entities. Our system exhibits robust performance in the MultiCoNER2 shared task, even in the low-resource language setting.
  arXiv  Detail & Related papers  (2023-04-20T20:30:34Z)
• Some Languages are More Equal than Others: Probing Deeper into the Linguistic Disparity in the NLP World [2.0777058026628583]
  Linguistic disparity in the NLP world is a problem that has been widely acknowledged recently. This paper provides a comprehensive analysis of the disparity that exists within the languages of the world.
  arXiv  Detail & Related papers  (2022-10-16T12:50:30Z)
• Systematic Inequalities in Language Technology Performance across the World's Languages [94.65681336393425]
  We introduce a framework for estimating the global utility of language technologies. Our analyses involve the field at large, but also more in-depth studies on both user-facing technologies and more linguistic NLP tasks.
  arXiv  Detail & Related papers  (2021-10-13T14:03:07Z)
• Debiasing Multilingual Word Embeddings: A Case Study of Three Indian Languages [9.208381487410191]
  We consider different methods to quantify bias and different debiasing approaches for monolingual as well as multilingual settings. Our proposed methods establish the state-of-the-art performance for debiasing multilingual embeddings for three Indian languages.
  arXiv  Detail & Related papers  (2021-07-21T16:12:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.