Datasets for Fairness in Language Models: An In-Depth Survey

• URL: http://arxiv.org/abs/2506.23411v1
• Date: Sun, 29 Jun 2025 22:11:58 GMT
• Title: Datasets for Fairness in Language Models: An In-Depth Survey
• Authors: Jiale Zhang, Zichong Wang, Avash Palikhe, Zhipeng Yin, Wenbin Zhang,
• Abstract summary: This survey examines the most widely used fairness datasets in current language model research.<n>We introduce a unified evaluation framework that reveals consistent patterns of demographic disparities across datasets and scoring methods.<n>We highlight the often overlooked biases that can influence conclusions about model fairness and offer practical guidance for selecting, combining, and interpreting these datasets.
• Score: 8.198294998446867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fairness benchmarks play a central role in shaping how we evaluate language models, yet surprisingly little attention has been given to examining the datasets that these benchmarks rely on. This survey addresses that gap by presenting a broad and careful review of the most widely used fairness datasets in current language model research, characterizing them along several key dimensions including their origin, scope, content, and intended use to help researchers better appreciate the assumptions and limitations embedded in these resources. To support more meaningful comparisons and analyses, we introduce a unified evaluation framework that reveals consistent patterns of demographic disparities across datasets and scoring methods. Applying this framework to twenty four common benchmarks, we highlight the often overlooked biases that can influence conclusions about model fairness and offer practical guidance for selecting, combining, and interpreting these datasets. We also point to opportunities for creating new fairness benchmarks that reflect more diverse social contexts and encourage more thoughtful use of these tools going forward. All code, data, and detailed results are publicly available at https://github.com/vanbanTruong/Fairness-in-Large-Language-Models/tree/main/datasets to promote transparency and reproducibility across the research community.

Related papers

• CALMA: A Process for Deriving Context-aligned Axes for Language Model Alignment [4.732046558763803]
  We introduce CALMA, a grounded, participatory methodology for eliciting context-relevant axes for evaluation and alignment.<n>Our findings demonstrate the value of evaluation practices based on open-ended and use-case-driven processes.
  arXiv  Detail & Related papers  (2025-07-11T22:33:11Z)
• PUB: Plot Understanding Benchmark and Dataset for Evaluating Large Language Models on Synthetic Visual Data Interpretation [2.1184929769291294]
  This paper presents a novel synthetic dataset designed to evaluate the proficiency of large language models in interpreting data visualizations. Our dataset is generated using controlled parameters to ensure comprehensive coverage of potential real-world scenarios. We employ multimodal text prompts with questions related to visual data in images to benchmark several state-of-the-art models.
  arXiv  Detail & Related papers  (2024-09-04T11:19:17Z)
• Do Text-to-Vis Benchmarks Test Real Use of Visualisations? [11.442971909006657]
  This paper investigates whether benchmarks reflect real-world use through an empirical study comparing benchmark datasets with code from public repositories. Our findings reveal a substantial gap, with evaluations not testing the same distribution of chart types, attributes, and actions as real-world examples. One dataset is representative, but requires extensive modification to become a practical end-to-end benchmark. This shows that new benchmarks are needed to support the development of systems that truly address users' visualisation needs.
  arXiv  Detail & Related papers  (2024-07-29T06:13:28Z)
• Data-Centric AI in the Age of Large Language Models [51.20451986068925]
  This position paper proposes a data-centric viewpoint of AI research, focusing on large language models (LLMs) We make the key observation that data is instrumental in the developmental (e.g., pretraining and fine-tuning) and inferential stages (e.g., in-context learning) of LLMs. We identify four specific scenarios centered around data, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
  arXiv  Detail & Related papers  (2024-06-20T16:34:07Z)
• The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representation [5.249002650134171]
  Large Language Models (LLMs) have emerged as powerful support tools across various natural language tasks and a range of application domains.<n>This paper provides a comparative overview of twelve studies investigating the potential of LLMs in labelling data.<n>While the models demonstrate promising cost and time-saving benefits, there exist considerable limitations, such as representativeness, bias, sensitivity to prompt variations and English language preference.
  arXiv  Detail & Related papers  (2024-05-02T14:00:22Z)
• Capture the Flag: Uncovering Data Insights with Large Language Models [90.47038584812925]
  This study explores the potential of using Large Language Models (LLMs) to automate the discovery of insights in data. We propose a new evaluation methodology based on a "capture the flag" principle, measuring the ability of such models to recognize meaningful and pertinent information (flags) in a dataset.
  arXiv  Detail & Related papers  (2023-12-21T14:20:06Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Perceptual Score: What Data Modalities Does Your Model Perceive? [73.75255606437808]
  We introduce the perceptual score, a metric that assesses the degree to which a model relies on the different subsets of the input features. We find that recent, more accurate multi-modal models for visual question-answering tend to perceive the visual data less than their predecessors. Using the perceptual score also helps to analyze model biases by decomposing the score into data subset contributions.
  arXiv  Detail & Related papers  (2021-10-27T12:19:56Z)
• Reliable Evaluations for Natural Language Inference based on a Unified Cross-dataset Benchmark [54.782397511033345]
  Crowd-sourced Natural Language Inference (NLI) datasets may suffer from significant biases like annotation artifacts. We present a new unified cross-datasets benchmark with 14 NLI datasets and re-evaluate 9 widely-used neural network-based NLI models. Our proposed evaluation scheme and experimental baselines could provide a basis to inspire future reliable NLI research.
  arXiv  Detail & Related papers  (2020-10-15T11:50:12Z)
• Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations [67.4375210552593]
  We design experiments to understand an important but often ignored problem in visually grounded language generation. Given that humans have different utilities and visual attention, how will the sample variance in multi-reference datasets affect the models' performance? We show that it is of paramount importance to report variance in experiments; that human-generated references could vary drastically in different datasets/tasks, revealing the nature of each task.
  arXiv  Detail & Related papers  (2020-10-07T20:45:14Z)
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models [6.998536937701312]
  Recent years have seen a growing number of publications that analyse Natural Language Inference (NLI) datasets for superficial cues. This structured survey provides an overview of the evolving research area by categorising reported weaknesses in models and datasets.
  arXiv  Detail & Related papers  (2020-05-29T17:55:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.