Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks

• URL: http://arxiv.org/abs/2304.03935v2
• Date: Fri, 14 Jul 2023 22:23:48 GMT
• Title: Last-Layer Fairness Fine-tuning is Simple and Effective for Neural Networks
• Authors: Yuzhen Mao, Zhun Deng, Huaxiu Yao, Ting Ye, Kenji Kawaguchi, James Zou
• Abstract summary: We develop a framework to train fair neural networks in an efficient and inexpensive way. Last-layer fine-tuning alone can effectively promote fairness in deep neural networks.
• Score: 36.182644157139144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As machine learning has been deployed ubiquitously across applications in modern data science, algorithmic fairness has become a great concern. Among them, imposing fairness constraints during learning, i.e. in-processing fair training, has been a popular type of training method because they don't require accessing sensitive attributes during test time in contrast to post-processing methods. While this has been extensively studied in classical machine learning models, their impact on deep neural networks remains unclear. Recent research has shown that adding fairness constraints to the objective function leads to severe over-fitting to fairness criteria in large models, and how to solve this challenge is an important open question. To tackle this, we leverage the wisdom and power of pre-training and fine-tuning and develop a simple but novel framework to train fair neural networks in an efficient and inexpensive way -- last-layer fine-tuning alone can effectively promote fairness in deep neural networks. This framework offers valuable insights into representation learning for training fair neural networks.

Related papers

• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Set-Based Training for Neural Network Verification [8.97708612393722]
  Small input perturbations can significantly affect the outputs of a neural network. In safety-critical environments, the inputs often contain noisy sensor data. We employ an end-to-end set-based training procedure that trains robust neural networks for formal verification.
  arXiv  Detail & Related papers  (2024-01-26T15:52:41Z)
• NeuralFastLAS: Fast Logic-Based Learning from Raw Data [54.938128496934695]
  Symbolic rule learners generate interpretable solutions, however they require the input to be encoded symbolically. Neuro-symbolic approaches overcome this issue by mapping raw data to latent symbolic concepts using a neural network. We introduce NeuralFastLAS, a scalable and fast end-to-end approach that trains a neural network jointly with a symbolic learner.
  arXiv  Detail & Related papers  (2023-10-08T12:33:42Z)
• FETA: Fairness Enforced Verifying, Training, and Predicting Algorithms for Neural Networks [9.967054059014691]
  We study the problem of verifying, training, and guaranteeing individual fairness of neural network models. A popular approach for enforcing fairness is to translate a fairness notion into constraints over the parameters of the model. We develop a counterexample-guided post-processing technique to provably enforce fairness constraints at prediction time.
  arXiv  Detail & Related papers  (2022-06-01T15:06:11Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Probabilistic Verification of Neural Networks Against Group Fairness [21.158245095699456]
  We propose an approach to formally verify neural networks against fairness. Our method is built upon an approach for learning Markov Chains from a user-provided neural network. We demonstrate that with our analysis results, the neural weights can be optimized to improve fairness.
  arXiv  Detail & Related papers  (2021-07-18T04:34:31Z)
• Technical Challenges for Training Fair Neural Networks [62.466658247995404]
  We conduct experiments on both facial recognition and automated medical diagnosis datasets using state-of-the-art architectures. We observe that large models overfit to fairness objectives, and produce a range of unintended and undesirable consequences.
  arXiv  Detail & Related papers  (2021-02-12T20:36:45Z)
• Provably Training Neural Network Classifiers under Fairness Constraints [70.64045590577318]
  We show that overparametrized neural networks could meet the constraints. Key ingredient of building a fair neural network classifier is establishing no-regret analysis for neural networks.
  arXiv  Detail & Related papers  (2020-12-30T18:46:50Z)
• A Principle of Least Action for the Training of Neural Networks [10.342408668490975]
  We show the presence of a low kinetic energy displacement bias in the transport map of the network, and link this bias with generalization performance. We propose a new learning algorithm, which automatically adapts to the complexity of the given task, and leads to networks with a high generalization ability even in low data regimes.
  arXiv  Detail & Related papers  (2020-09-17T15:37:34Z)
• Feature Purification: How Adversarial Training Performs Robust Deep Learning [66.05472746340142]
  We show a principle that we call Feature Purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly gradient descent indeed this principle.
  arXiv  Detail & Related papers  (2020-05-20T16:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.