Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES)

• URL: http://arxiv.org/abs/2510.24958v1
• Date: Tue, 28 Oct 2025 20:42:14 GMT
• Title: Adaptive Data Collection for Latin-American Community-sourced Evaluation of Stereotypes (LACES)
• Authors: Guido Ivetta, Pietro Palombini, Sofía Martinelli, Marcos J Gomez, Sunipa Dev, Vinodkumar Prabhakaran, Luciana Benotti,
• Abstract summary: The evaluation of societal biases in NLP models is critically hindered by a glaring geo-cultural gap.<n>Existing benchmarks are overwhelmingly English-centric and focused on U.S. demographics.<n>We introduce a new, large-scale dataset of stereotypes developed through targeted community partnerships within Latin America.
• Score: 12.636379779655558
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The evaluation of societal biases in NLP models is critically hindered by a glaring geo-cultural gap, as existing benchmarks are overwhelmingly English-centric and focused on U.S. demographics. This leaves regions such as Latin America severely underserved, making it impossible to adequately assess or mitigate the perpetuation of harmful regional stereotypes by language technologies. To address this gap, we introduce a new, large-scale dataset of stereotypes developed through targeted community partnerships within Latin America. Furthermore, we present a novel dynamic data collection methodology that uniquely integrates the sourcing of new stereotype entries and the validation of existing data within a single, unified workflow. This combined approach results in a resource with significantly broader coverage and higher regional nuance than static collection methods. We believe that this new method could be applicable in gathering sociocultural knowledge of other kinds, and that this dataset provides a crucial new resource enabling robust stereotype evaluation and significantly addressing the geo-cultural deficit in fairness resources for Latin America.

Related papers

• SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Context [10.43559852429736]
  Stereotype repositories are critical to assess generative AI model safety, but currently lack adequate global coverage.<n>This work introduces a multilingual stereotype resource covering four sub-Saharan African countries that are severely underrepresented in NLP resources: Ghana, Kenya, Nigeria, and South Africa.
  arXiv  Detail & Related papers  (2026-02-25T20:56:27Z)
• Challenging the Abilities of Large Language Models in Italian: a Community Initiative [63.94242079171895]
  "Challenging the Abilities of LAnguage Models in ITAlian" (CALAMITA) is a large-scale collaborative benchmarking initiative for Italian.<n>It federates more than 80 contributors from academia, industry, and the public sector to design, document, and evaluate a diverse collection of tasks.<n>We report results for four open-weight LLMs, highlighting systematic strengths and weaknesses across abilities.
  arXiv  Detail & Related papers  (2025-12-04T12:50:29Z)
• SESGO: Spanish Evaluation of Stereotypical Generative Outputs [1.1549572298362782]
  This paper addresses the critical gap in evaluating bias in multilingual Large Language Models (LLMs)<n>Current evaluations remain predominantly US-English-centric, leaving potential harms in other linguistic and cultural contexts largely underexamined.<n>We introduce a novel, culturally-grounded framework for detecting social biases in instruction-tuned LLMs.
  arXiv  Detail & Related papers  (2025-09-03T14:04:51Z)
• ValuesRAG: Enhancing Cultural Alignment Through Retrieval-Augmented Contextual Learning [1.1343849658875087]
  ValuesRAG is a novel framework that integrates cultural and demographic knowledge dynamically during text generation.<n>We evaluate ValuesRAG using 6 diverse regional datasets and show that it consistently outperforms baselines.<n>Our findings underscore the potential of dynamic retrieval-based methods to bridge the gap between global LLM capabilities and localized cultural values.
  arXiv  Detail & Related papers  (2025-01-02T03:26:13Z)
• No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models [38.932610459192105]
  We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs) Our work underscores the value of using diverse data to create more inclusive multimodal systems.
  arXiv  Detail & Related papers  (2024-05-22T16:04:22Z)
• Leveraging Prototypical Representations for Mitigating Social Bias without Demographic Information [50.29934517930506]
  DAFair is a novel approach to address social bias in language models. We leverage prototypical demographic texts and incorporate a regularization term during the fine-tuning process to mitigate bias.
  arXiv  Detail & Related papers  (2024-03-14T15:58:36Z)
• SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes [18.991295993710224]
  SeeGULL is a global-scale multilingual dataset of social stereotypes, spanning 20 languages, with human annotations across 23 regions, and demonstrate its utility in identifying gaps in model evaluations.
  arXiv  Detail & Related papers  (2024-03-08T22:09:58Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• Survey of Social Bias in Vision-Language Models [65.44579542312489]
  Survey aims to provide researchers with a high-level insight into the similarities and differences of social bias studies in pre-trained models across NLP, CV, and VL. The findings and recommendations presented here can benefit the ML community, fostering the development of fairer and non-biased AI models.
  arXiv  Detail & Related papers  (2023-09-24T15:34:56Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Towards Understanding and Mitigating Social Biases in Language Models [107.82654101403264]
  Large-scale pretrained language models (LMs) can be potentially dangerous in manifesting undesirable representational biases. We propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information.
  arXiv  Detail & Related papers  (2021-06-24T17:52:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.