Understanding the Effect of Model Compression on Social Bias in Large Language Models

• URL: http://arxiv.org/abs/2312.05662v2
• Date: Tue, 12 Dec 2023 12:51:52 GMT
• Title: Understanding the Effect of Model Compression on Social Bias in Large Language Models
• Authors: Gustavo Gon\c{c}alves and Emma Strubell
• Abstract summary: Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. We study the impact of model compression via quantization and knowledge distillation on measures of social bias in LLMs.
• Score: 12.289003145872481
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tasks, leading to representational harm. Many strategies have been proposed to mitigate the effects of inappropriate social biases learned during pretraining. Simultaneously, methods for model compression have become increasingly popular to reduce the computational burden of LLMs. Despite the popularity and need for both approaches, little work has been done to explore the interplay between these two. We perform a carefully controlled study of the impact of model compression via quantization and knowledge distillation on measures of social bias in LLMs. Longer pretraining and larger models led to higher social bias, and quantization showed a regularizer effect with its best trade-off around 20% of the original pretraining time.

Related papers

• Social Debiasing for Fair Multi-modal LLMs [55.8071045346024]
  Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities. However, these models often inherit severe social biases from their training datasets, leading to unfair predictions based on attributes like race and gender. This paper addresses the issue of social biases in MLLMs by i) Introducing a comprehensive Counterfactual dataset with Multiple Social Concepts (CMSC) and ii) Proposing an Anti-Stereotype Debiasing strategy (ASD)
  arXiv  Detail & Related papers  (2024-08-13T02:08:32Z)
• Understanding the Interplay of Scale, Data, and Bias in Language Models: A Case Study with BERT [4.807994469764776]
  We study the influence of model scale and pre-training data on a language model's learnt social biases. Our experiments show that pre-training data substantially influences how upstream biases evolve with model scale. We shed light on the complex interplay of data and model scale, and investigate how it translates to concrete biases.
  arXiv  Detail & Related papers  (2024-07-25T23:09:33Z)
• The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models [78.69526166193236]
  Pre-trained Language models (PLMs) have been acknowledged to contain harmful information, such as social biases. We propose sc Social Bias Neurons to accurately pinpoint units (i.e., neurons) in a language model that can be attributed to undesirable behavior, such as social bias. As measured by prior metrics from StereoSet, our model achieves a higher degree of fairness while maintaining language modeling ability with low cost.
  arXiv  Detail & Related papers  (2024-06-14T15:41:06Z)
• On the social bias of speech self-supervised models [45.787612513520386]
  Social bias in SSL models can perpetuate injustice by automating discriminatory patterns and reinforcing inequitable systems. We probe how various factors, such as model architecture, size, and training methodologies, influence the propagation of social bias within these models. Our findings reveal that employing techniques such as row-pruning and training wider, shallower models can effectively mitigate social bias within SSL model.
  arXiv  Detail & Related papers  (2024-06-07T15:07:07Z)
• Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecasting [74.68371461260946]
  SocialSense is a framework that induces a belief-centered graph on top of an existent social network, along with graph-based propagation to capture social dynamics. Our method surpasses existing state-of-the-art in experimental evaluations for both zero-shot and supervised settings.
  arXiv  Detail & Related papers  (2023-10-20T06:17:02Z)
• Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition [99.7047087527422]
  In this work, we demonstrate that competition can fundamentally alter the behavior of machine learning scaling trends. We find many settings where improving data representation quality decreases the overall predictive accuracy across users. At a conceptual level, our work suggests that favorable scaling trends for individual model-providers need not translate to downstream improvements in social welfare.
  arXiv  Detail & Related papers  (2023-06-26T13:06:34Z)
• Soft Attention: Does it Actually Help to Learn Social Interactions in Pedestrian Trajectory Prediction? [2.180763067449862]
  We consider the problem of predicting the future path of a pedestrian using its motion history and the motion history of the surrounding pedestrians. Deep-learning has become the main tool used to model the impact of social interactions on a pedestrian's motion.
  arXiv  Detail & Related papers  (2021-06-16T17:39:35Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.