Who's the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration

• URL: http://arxiv.org/abs/2412.10575v2
• Date: Mon, 14 Apr 2025 19:40:08 GMT
• Title: Who's the (Multi-)Fairest of Them All: Rethinking Interpolation-Based Data Augmentation Through the Lens of Multicalibration
• Authors: Karina Halevy, Karly Hou, Charumathi Badrinath,
• Abstract summary: We stress-test four versions of Fair Mixup on two structured data classification problems with up to 81 marginalized groups.<n>We find that on nearly every experiment, Fair Mixup textitworsens baseline performance and fairness, but the simple vanilla Mixup textitoutperforms both Fair Mixup and the baseline.
• Score: 1.0923877073891446
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data augmentation methods, especially SoTA interpolation-based methods such as Fair Mixup, have been widely shown to increase model fairness. However, this fairness is evaluated on metrics that do not capture model uncertainty and on datasets with only one, relatively large, minority group. As a remedy, multicalibration has been introduced to measure fairness while accommodating uncertainty and accounting for multiple minority groups. However, existing methods of improving multicalibration involve reducing initial training data to create a holdout set for post-processing, which is not ideal when minority training data is already sparse. This paper uses multicalibration to more rigorously examine data augmentation for classification fairness. We stress-test four versions of Fair Mixup on two structured data classification problems with up to 81 marginalized groups, evaluating multicalibration violations and balanced accuracy. We find that on nearly every experiment, Fair Mixup \textit{worsens} baseline performance and fairness, but the simple vanilla Mixup \textit{outperforms} both Fair Mixup and the baseline, especially when calibrating on small groups. \textit{Combining} vanilla Mixup with multicalibration post-processing, which enforces multicalibration through post-processing on a holdout set, further increases fairness.

Related papers

• Mix from Failure: Confusion-Pairing Mixup for Long-Tailed Recognition [14.009773753739282]
  Long-tailed image recognition is a problem considering a real-world class distribution rather than an artificial uniform. In this paper, we tackle the problem from a different perspective to augment a training dataset to enhance the sample diversity of minority classes. Our method, namely Confusion-Pairing Mixup (CP-Mix), estimates the confusion distribution of the model and handles the data deficiency problem.
  arXiv  Detail & Related papers  (2024-11-12T08:08:31Z)
• Data Augmentation via Subgroup Mixup for Improving Fairness [31.296907816698987]
  We propose data augmentation via pairwise mixup across subgroups to improve group fairness. Inspired by the successes of mixup for improving classification performance, we develop a pairwise mixup scheme to augment training data.
  arXiv  Detail & Related papers  (2023-09-13T17:32:21Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Generative Oversampling for Imbalanced Data via Majority-Guided VAE [15.93867386081279]
  We propose a novel over-sampling model, called Majority-Guided VAE(MGVAE), which generates new minority samples under the guidance of a majority-based prior. In this way, the newly generated minority samples can inherit the diversity and richness of the majority ones, thus mitigating overfitting in downstream tasks.
  arXiv  Detail & Related papers  (2023-02-14T06:35:23Z)
• Learning Informative Representation for Fairness-aware Multivariate Time-series Forecasting: A Group-based Perspective [50.093280002375984]
  Performance unfairness among variables widely exists in multivariate time series (MTS) forecasting models. We propose a novel framework, named FairFor, for fairness-aware MTS forecasting.
  arXiv  Detail & Related papers  (2023-01-27T04:54:12Z)
• C-Mixup: Improving Generalization in Regression [71.10418219781575]
  Mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. We propose C-Mixup, which adjusts the sampling probability based on the similarity of the labels. C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively.
  arXiv  Detail & Related papers  (2022-10-11T20:39:38Z)
• Bias Mimicking: A Simple Sampling Approach for Bias Mitigation [57.17709477668213]
  We introduce a new class-conditioned sampling method: Bias Mimicking. Bias Mimicking improves underrepresented groups' accuracy of sampling methods by 3% over four benchmarks.
  arXiv  Detail & Related papers  (2022-09-30T17:33:00Z)
• Fair admission risk prediction with proportional multicalibration [0.16249424686052708]
  Multicalibration constrains calibration error among flexibly-defined subpopulations. It is possible for a decision-maker to learn to trust or distrust model predictions for specific groups. We propose proportional multicalibration, a criteria that constrains the percent calibration error among groups and within prediction bins.
  arXiv  Detail & Related papers  (2022-09-29T08:15:29Z)
• How Robust is Your Fairness? Evaluating and Sustaining Fairness under Unseen Distribution Shifts [107.72786199113183]
  We propose a novel fairness learning method termed CUrvature MAtching (CUMA) CUMA achieves robust fairness generalizable to unseen domains with unknown distributional shifts. We evaluate our method on three popular fairness datasets.
  arXiv  Detail & Related papers  (2022-07-04T02:37:50Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective [17.733087434470907]
  Real-world data universally confronts a severe class-imbalance problem and exhibits a long-tailed distribution. We propose two novel methods from the prior perspective to alleviate this dilemma. First, we deduce a balance-oriented data augmentation named Uniform Mixup (UniMix) to promote mixup in long-tailed scenarios. Second, motivated by the Bayesian theory, we figure out the Bayes Bias (Bayias) to compensate it as a modification on standard cross-entropy loss.
  arXiv  Detail & Related papers  (2021-11-06T12:53:34Z)
• Fair Mixup: Fairness via Interpolation [28.508444261249423]
  We propose fair mixup, a new data augmentation strategy for imposing the fairness constraint. We show that fairness can be achieved by regularizing the models on paths of interpolated samples between the groups. We empirically show that it ensures a better generalization for both accuracy and fairness measurement in benchmarks.
  arXiv  Detail & Related papers  (2021-03-11T06:57:26Z)
• M2m: Imbalanced Classification via Major-to-minor Translation [79.09018382489506]
  In most real-world scenarios, labeled training datasets are highly class-imbalanced, where deep neural networks suffer from generalizing to a balanced testing criterion. In this paper, we explore a novel yet simple way to alleviate this issue by augmenting less-frequent classes via translating samples from more-frequent classes. Our experimental results on a variety of class-imbalanced datasets show that the proposed method improves the generalization on minority classes significantly compared to other existing re-sampling or re-weighting methods.
  arXiv  Detail & Related papers  (2020-04-01T13:21:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.