Balancing Act: Constraining Disparate Impact in Sparse Models
- URL: http://arxiv.org/abs/2310.20673v2
- Date: Fri, 8 Mar 2024 00:22:38 GMT
- Title: Balancing Act: Constraining Disparate Impact in Sparse Models
- Authors: Meraj Hashemizadeh, Juan Ramirez, Rohan Sukumaran, Golnoosh Farnadi,
Simon Lacoste-Julien, Jose Gallego-Posada
- Abstract summary: We propose a constrained optimization approach that directly addresses the disparate impact of pruning.
Our formulation bounds the accuracy change between the dense and sparse models, for each sub-group.
Experimental results demonstrate that our technique scales reliably to problems involving large models and hundreds of protected sub-groups.
- Score: 20.058720715290434
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Model pruning is a popular approach to enable the deployment of large deep
learning models on edge devices with restricted computational or storage
capacities. Although sparse models achieve performance comparable to that of
their dense counterparts at the level of the entire dataset, they exhibit high
accuracy drops for some data sub-groups. Existing methods to mitigate this
disparate impact induced by pruning (i) rely on surrogate metrics that address
the problem indirectly and have limited interpretability; or (ii) scale poorly
with the number of protected sub-groups in terms of computational cost. We
propose a constrained optimization approach that directly addresses the
disparate impact of pruning: our formulation bounds the accuracy change between
the dense and sparse models, for each sub-group. This choice of constraints
provides an interpretable success criterion to determine if a pruned model
achieves acceptable disparity levels. Experimental results demonstrate that our
technique scales reliably to problems involving large models and hundreds of
protected sub-groups.
Related papers
- Precise Model Benchmarking with Only a Few Observations [6.092112060364272]
We propose an empirical Bayes (EB) estimator that balances direct and regression estimates for each subgroup separately.
EB consistently provides more precise estimates of the LLM performance compared to the direct and regression approaches.
arXiv Detail & Related papers (2024-10-07T17:26:31Z) - Querying Easily Flip-flopped Samples for Deep Active Learning [63.62397322172216]
Active learning is a machine learning paradigm that aims to improve the performance of a model by strategically selecting and querying unlabeled data.
One effective selection strategy is to base it on the model's predictive uncertainty, which can be interpreted as a measure of how informative a sample is.
This paper proposes the it least disagree metric (LDM) as the smallest probability of disagreement of the predicted label.
arXiv Detail & Related papers (2024-01-18T08:12:23Z) - Variable Importance Matching for Causal Inference [73.25504313552516]
We describe a general framework called Model-to-Match that achieves these goals.
Model-to-Match uses variable importance measurements to construct a distance metric.
We operationalize the Model-to-Match framework with LASSO.
arXiv Detail & Related papers (2023-02-23T00:43:03Z) - Variational Inference with Locally Enhanced Bounds for Hierarchical
Models [38.73307745906571]
We propose a new family of variational bounds for hierarchical models based on the application of tightening methods.
We show that our approach naturally allows the use of subsampling to get unbiased gradients, and that it fully leverages the power of methods that build tighter lower bounds.
arXiv Detail & Related papers (2022-03-08T22:53:43Z) - Automatic Mixed-Precision Quantization Search of BERT [62.65905462141319]
Pre-trained language models such as BERT have shown remarkable effectiveness in various natural language processing tasks.
These models usually contain millions of parameters, which prevents them from practical deployment on resource-constrained devices.
We propose an automatic mixed-precision quantization framework designed for BERT that can simultaneously conduct quantization and pruning in a subgroup-wise level.
arXiv Detail & Related papers (2021-12-30T06:32:47Z) - Characterizing Fairness Over the Set of Good Models Under Selective
Labels [69.64662540443162]
We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance.
We provide tractable algorithms to compute the range of attainable group-level predictive disparities.
We extend our framework to address the empirically relevant challenge of selectively labelled data.
arXiv Detail & Related papers (2021-01-02T02:11:37Z) - MASSIVE: Tractable and Robust Bayesian Learning of Many-Dimensional
Instrumental Variable Models [8.271859911016719]
We propose a general and efficient causal inference algorithm that accounts for model uncertainty.
We show that, as long as some of the candidates are (close to) valid, without knowing a priori which ones, they collectively still pose enough restrictions on the target interaction to obtain a reliable causal effect estimate.
arXiv Detail & Related papers (2020-12-18T10:06:55Z) - Learning Consistent Deep Generative Models from Sparse Data via
Prediction Constraints [16.48824312904122]
We develop a new framework for learning variational autoencoders and other deep generative models.
We show that these two contributions -- prediction constraints and consistency constraints -- lead to promising image classification performance.
arXiv Detail & Related papers (2020-12-12T04:18:50Z) - Efficient Ensemble Model Generation for Uncertainty Estimation with
Bayesian Approximation in Segmentation [74.06904875527556]
We propose a generic and efficient segmentation framework to construct ensemble segmentation models.
In the proposed method, ensemble models can be efficiently generated by using the layer selection method.
We also devise a new pixel-wise uncertainty loss, which improves the predictive performance.
arXiv Detail & Related papers (2020-05-21T16:08:38Z) - Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples.
We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries.
We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
arXiv Detail & Related papers (2020-02-27T10:22:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.