Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning

• URL: http://arxiv.org/abs/2402.00910v2
• Date: Tue, 13 Feb 2024 18:54:59 GMT
• Title: Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning
• Authors: Ahmed Radwan, Layan Zaafarani, Jetana Abudawood, Faisal AlZahrani, Fares Fourati
• Abstract summary: This paper proposes a comprehensive approach using multiple methods to remove bias in AI models. We train multiple models with the counter-bias of the pre-trained model through data splitting, local training, and regularized fine-tuning. We conclude our solution with knowledge distillation that results in a single unbiased neural network.
• Score: 0.2812395851874055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Addressing biases in AI models is crucial for ensuring fair and accurate predictions. However, obtaining large, unbiased datasets for training can be challenging. This paper proposes a comprehensive approach using multiple methods to remove bias in AI models, with only a small dataset and a potentially biased pretrained model. We train multiple models with the counter-bias of the pre-trained model through data splitting, local training, and regularized fine-tuning, gaining potentially counter-biased models. Then, we employ ensemble learning for all models to reach unbiased predictions. To further accelerate the inference time of our ensemble model, we conclude our solution with knowledge distillation that results in a single unbiased neural network. We demonstrate the effectiveness of our approach through experiments on the CIFAR10 and HAM10000 datasets, showcasing promising results. This work contributes to the ongoing effort to create more unbiased and reliable AI models, even with limited data availability.

Related papers

• Improving Bias Mitigation through Bias Experts in Natural Language Understanding [10.363406065066538]
  We propose a new debiasing framework that introduces binary classifiers between the auxiliary model and the main model. Our proposed strategy improves the bias identification ability of the auxiliary model.
  arXiv  Detail & Related papers  (2023-12-06T16:15:00Z)
• Debiasing Multimodal Models via Causal Information Minimization [65.23982806840182]
  We study bias arising from confounders in a causal graph for multimodal data. Robust predictive features contain diverse information that helps a model generalize to out-of-distribution data. We use these features as confounder representations and use them via methods motivated by causal theory to remove bias from models.
  arXiv  Detail & Related papers  (2023-11-28T16:46:14Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Fast Model Debias with Machine Unlearning [54.32026474971696]
  Deep neural networks might behave in a biased manner in many real-world scenarios. Existing debiasing methods suffer from high costs in bias labeling or model re-training. We propose a fast model debiasing framework (FMD) which offers an efficient approach to identify, evaluate and remove biases.
  arXiv  Detail & Related papers  (2023-10-19T08:10:57Z)
• Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biases [5.997909991352044]
  Recent work sought to develop robust, unbiased models by filtering biased examples from training sets. We argue that such filtering can obscure the true capabilities of models to overcome biases. We introduce an evaluation framework defined by a bias-amplified training set and an anti-biased test set.
  arXiv  Detail & Related papers  (2023-05-30T10:10:42Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning [92.89846887298852]
  Consider making a prediction over new test data without any opportunity to learn from a training set of labelled data. Give access to a set of expert models and their predictions alongside some limited information about the dataset used to train them.
  arXiv  Detail & Related papers  (2022-10-11T10:20:31Z)
• Toward More Generalized Malicious URL Detection Models [4.151658495779136]
  This paper reveals a data bias issue that can severely affect the performance while conducting a machine learning model for malicious URL detection. We describe how such bias can be identified using interpretable machine learning techniques, and argue that such biases naturally exist in the real world security data for training a classification model. We propose a debiased training strategy that can be applied to most deep-learning based models to alleviate the negative effects from the biased features.
  arXiv  Detail & Related papers  (2022-02-21T07:46:03Z)
• Learning from others' mistakes: Avoiding dataset biases without modeling them [111.17078939377313]
  State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended task. Previous work has demonstrated effective methods to circumvent these issues when knowledge of the bias is available. We show a method for training models that learn to ignore these problematic correlations.
  arXiv  Detail & Related papers  (2020-12-02T16:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.