The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models

• URL: http://arxiv.org/abs/2405.04623v1
• Date: Tue, 7 May 2024 19:11:10 GMT
• Title: The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal Models
• Authors: Abeba Birhane, Sepehr Dehdashtian, Vinay Uday Prabhu, Vishnu Boddeti,
• Abstract summary: We evaluate the downstream impact of dataset scaling on visio-linguistic models trained on the LAION400-M and LAION-2B datasets. Our results show that as the training data increased, the probability of a pre-trained CLIP model misclassifying human images increased. For the smaller base ViT-B models, the probability of predicting an image of a Black man and a Latino man as criminal decreases by 20% and 47%, respectively, when the dataset is scaled from 400M to 2B samples.
• Score: 1.6076959385522371
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Scale the model, scale the data, scale the GPU farms is the reigning sentiment in the world of generative AI today. While model scaling has been extensively studied, data scaling and its downstream impacts on model performance remain under-explored. This is particularly important in the context of multimodal datasets whose main source is the World Wide Web, condensed and packaged as the Common Crawl dump, which is known to exhibit numerous drawbacks. In this paper, we evaluate the downstream impact of dataset scaling on 14 visio-linguistic models (VLMs) trained on the LAION400-M and LAION-2B datasets by measuring racial and gender bias using the Chicago Face Dataset (CFD) as the probe. Our results show that as the training data increased, the probability of a pre-trained CLIP model misclassifying human images as offensive non-human classes such as chimpanzee, gorilla, and orangutan decreased, but misclassifying the same images as human offensive classes such as criminal increased. Furthermore, of the 14 Vision Transformer-based VLMs we evaluated, the probability of predicting an image of a Black man and a Latino man as criminal increases by 65% and 69%, respectively, when the dataset is scaled from 400M to 2B samples for the larger ViT-L models. Conversely, for the smaller base ViT-B models, the probability of predicting an image of a Black man and a Latino man as criminal decreases by 20% and 47%, respectively, when the dataset is scaled from 400M to 2B samples. We ground the model audit results in a qualitative and historical analysis, reflect on our findings and their implications for dataset curation practice, and close with a summary of mitigation mechanisms and ways forward. Content warning: This article contains racially dehumanising and offensive descriptions.

Related papers

• Does Data-Efficient Generalization Exacerbate Bias in Foundation Models? [2.298227866545911]
  Foundation models have emerged as robust models with label efficiency in diverse domains. It is unclear whether using a large amount of unlabeled data, biased by the presence of sensitive attributes during pre-training, influences the fairness of the model. This research examines the bias in the Foundation model when it is applied to fine-tune the Brazilian Multilabel Ophthalmological dataset.
  arXiv  Detail & Related papers  (2024-08-28T22:14:44Z)
• Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI [17.101569078791492]
  We study 43 CLIP vision-language models to determine whether they learn human-like facial impression biases. We show for the first time that the the degree to which a bias is shared across a society predicts the degree to which it is reflected in a CLIP model.
  arXiv  Detail & Related papers  (2024-08-04T08:26:58Z)
• Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT [4.807994469764776]
  We study the influence of model scale and pre-training data on a language model's learnt social biases. Our experiments show that pre-training data substantially influences how upstream biases evolve with model scale. We shed light on the complex interplay of data and model scale, and investigate how it translates to concrete biases.
  arXiv  Detail & Related papers  (2024-07-25T23:09:33Z)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• Into the LAIONs Den: Investigating Hate in Multimodal Datasets [67.21783778038645]
  This paper investigates the effect of scaling datasets on hateful content through a comparative audit of two datasets: LAION-400M and LAION-2B. We found that hate content increased by nearly 12% with dataset scale, measured both qualitatively and quantitatively. We also found that filtering dataset contents based on Not Safe For Work (NSFW) values calculated based on images alone does not exclude all the harmful content in alt-text.
  arXiv  Detail & Related papers  (2023-11-06T19:00:05Z)
• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• Scaling Laws Do Not Scale [54.72120385955072]
  Recent work has argued that as the size of a dataset increases, the performance of a model trained on that dataset will increase. We argue that this scaling law relationship depends on metrics used to measure performance that may not correspond with how different groups of people perceive the quality of models' output. Different communities may also have values in tension with each other, leading to difficult, potentially irreconcilable choices about metrics used for model evaluations.
  arXiv  Detail & Related papers  (2023-07-05T15:32:21Z)
• On Hate Scaling Laws For Data-Swamps [14.891493485229251]
  We show that the presence of hateful content in datasets, when measured with a Hate Content Rate (HCR) metric, increased by nearly $12%$. As scale increased, the tendency of the model to associate images of human faces with the human being' class over 7 other offensive classes reduced by half. For the Black female category, the tendency of the model to associate their faces with the criminal' class doubled, while quintupling for Black male faces.
  arXiv  Detail & Related papers  (2023-06-22T18:00:17Z)
• Inverse Scaling: When Bigger Isn't Better [80.42834197416444]
  Large language models (LMs) show predictable improvements to overall loss with increased scale. We present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale.
  arXiv  Detail & Related papers  (2023-06-15T20:11:23Z)
• Quantifying Human Bias and Knowledge to guide ML models during Training [0.0]
  We introduce an experimental approach to dealing with skewed datasets by including humans in the training process. We ask humans to rank the importance of features of the dataset, and through rank aggregation, determine the initial weight bias for the model. We show that collective human bias can allow ML models to learn insights about the true population instead of the biased sample.
  arXiv  Detail & Related papers  (2022-11-19T20:49:07Z)
• On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study [65.17429512679695]
  In adversarial data collection (ADC), a human workforce interacts with a model in real time, attempting to produce examples that elicit incorrect predictions. Despite ADC's intuitive appeal, it remains unclear when training on adversarial datasets produces more robust models.
  arXiv  Detail & Related papers  (2021-06-02T00:48:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.