Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective

• URL: http://arxiv.org/abs/2312.10181v1
• Date: Fri, 15 Dec 2023 20:08:53 GMT
• Title: Coupling Fairness and Pruning in a Single Run: a Bi-level Optimization Perspective
• Authors: Yucong Dai, Gen Li, Feng Luo, Xiaolong Ma, Yongkai Wu
• Abstract summary: We propose a framework to jointly optimize the pruning mask and weight update processes with fairness constraints. This framework is engineered to compress models that maintain performance while ensuring fairness in a single execution. Our empirical analysis contrasts our framework with several mainstream pruning strategies, emphasizing our method's superiority in maintaining model fairness, performance, and efficiency.
• Score: 17.394732703591462
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks have demonstrated remarkable performance in various tasks. With a growing need for sparse deep learning, model compression techniques, especially pruning, have gained significant attention. However, conventional pruning techniques can inadvertently exacerbate algorithmic bias, resulting in unequal predictions. To address this, we define a fair pruning task where a sparse model is derived subject to fairness requirements. In particular, we propose a framework to jointly optimize the pruning mask and weight update processes with fairness constraints. This framework is engineered to compress models that maintain performance while ensuring fairness in a single execution. To this end, we formulate the fair pruning problem as a novel constrained bi-level optimization task and derive efficient and effective solving strategies. We design experiments spanning various datasets and settings to validate our proposed method. Our empirical analysis contrasts our framework with several mainstream pruning strategies, emphasizing our method's superiority in maintaining model fairness, performance, and efficiency.

Related papers

• A First-order Generative Bilevel Optimization Framework for Diffusion Models [57.40597004445473]
  Diffusion models iteratively denoise data samples to synthesize high-quality outputs. Traditional bilevel methods fail due to infinite-dimensional probability space and prohibitive sampling costs. We formalize this challenge as a generative bilevel optimization problem. Our first-order bilevel framework overcomes the incompatibility of conventional bilevel methods with diffusion processes.
  arXiv  Detail & Related papers  (2025-02-12T21:44:06Z)
• Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent [74.02034188307857]
  Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data. We find existing methods inevitably discard task-specific information that, while causing conflicts, is crucial for performance. Our approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.
  arXiv  Detail & Related papers  (2025-01-02T12:45:21Z)
• Lossless Model Compression via Joint Low-Rank Factorization Optimization [3.318320512635214]
  Low-rank factorization is a popular model compression technique that minimizes the error $delta$ between approximated and original weight matrices. Despite achieving performances close to the original models when $delta$ is optimized, a performance discrepancy remains due to the separate optimization processes for low-rank factorization and model performance. We introduce a novel joint optimization strategy for lossless low-rank weight factorization, which, for the first time, enhances the model's performance beyond the original.
  arXiv  Detail & Related papers  (2024-12-09T09:37:54Z)
• Selective Mixup Fine-Tuning for Optimizing Non-Decomposable Objectives [17.10165955576643]
  Current state-of-the-art empirical techniques offer sub-optimal performance on practical, non-decomposable performance objectives. We propose SelMix, a selective mixup-based inexpensive fine-tuning technique for pre-trained models. We find that proposed SelMix fine-tuning significantly improves the performance for various practical non-decomposable objectives across benchmarks.
  arXiv  Detail & Related papers  (2024-03-27T06:55:23Z)
• Diffusion Model for Data-Driven Black-Box Optimization [54.25693582870226]
  We focus on diffusion models, a powerful generative AI technology, and investigate their potential for black-box optimization. We study two practical types of labels: 1) noisy measurements of a real-valued reward function and 2) human preference based on pairwise comparisons. Our proposed method reformulates the design optimization problem into a conditional sampling problem, which allows us to leverage the power of diffusion models.
  arXiv  Detail & Related papers  (2024-03-20T00:41:12Z)
• Refined Coreset Selection: Towards Minimal Coreset Size under Model Performance Constraints [69.27190330994635]
  Coreset selection is powerful in reducing computational costs and accelerating data processing for deep learning algorithms. We propose an innovative method, which maintains optimization priority order over the model performance and coreset size. Empirically, extensive experiments confirm its superiority, often yielding better model performance with smaller coreset sizes.
  arXiv  Detail & Related papers  (2023-11-15T03:43:04Z)
• Pseudo-Bayesian Optimization [7.556071491014536]
  We study an axiomatic framework that elicits the minimal requirements to guarantee black-box optimization convergence. We show how using simple local regression, and a suitable "randomized prior" construction to quantify uncertainty, not only guarantees convergence but also consistently outperforms state-of-the-art benchmarks.
  arXiv  Detail & Related papers  (2023-10-15T07:55:28Z)
• Gradient constrained sharpness-aware prompt learning for vision-language models [99.74832984957025]
  This paper targets a novel trade-off problem in generalizable prompt learning for vision-language models (VLM) By analyzing the loss landscapes of the state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based method, we conclude that the trade-off performance correlates to both loss value and loss sharpness. We propose a novel SAM-based method for prompt learning, denoted as Gradient Constrained Sharpness-aware Context Optimization (GCSCoOp)
  arXiv  Detail & Related papers  (2023-09-14T17:13:54Z)
• Towards Optimal Compression: Joint Pruning and Quantization [1.191194620421783]
  This paper introduces FITCompress, a novel method integrating layer-wise mixed-precision quantization and unstructured pruning. Experiments on computer vision and natural language processing benchmarks demonstrate that our proposed approach achieves a superior compression-performance trade-off.
  arXiv  Detail & Related papers  (2023-02-15T12:02:30Z)
• Adapting by Pruning: A Case Study on BERT [9.963251767416967]
  We propose a novel model adaptation paradigm, adapting by pruning, which prunes neural connections in the pre-trained model to optimise the performance on the target task. We formulate adapting-by-pruning as an optimisation problem with a differentiable loss and propose an efficient algorithm to prune the model. Results suggest that our method can prune up to 50% weights in BERT while yielding similar performance compared to the fine-tuned full model.
  arXiv  Detail & Related papers  (2021-05-07T15:51:08Z)
• Beyond Individual and Group Fairness [90.4666341812857]
  We present a new data-driven model of fairness that is guided by the unfairness complaints received by the system. Our model supports multiple fairness criteria and takes into account their potential incompatibilities.
  arXiv  Detail & Related papers  (2020-08-21T14:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.