How does overparametrization affect performance on minority groups?

• URL: http://arxiv.org/abs/2206.03515v1
• Date: Tue, 7 Jun 2022 18:00:52 GMT
• Title: How does overparametrization affect performance on minority groups?
• Authors: Subha Maity, Saptarshi Roy, Songkai Xue, Mikhail Yurochkin, Yuekai Sun
• Abstract summary: We show that over parameterization always improves minority group performance. In a setting in which the regression functions for the majority and minority groups are different, we show that over parameterization always improves minority group performance.
• Score: 39.54853544590893
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The benefits of overparameterization for the overall performance of modern machine learning (ML) models are well known. However, the effect of overparameterization at a more granular level of data subgroups is less understood. Recent empirical studies demonstrate encouraging results: (i) when groups are not known, overparameterized models trained with empirical risk minimization (ERM) perform better on minority groups; (ii) when groups are known, ERM on data subsampled to equalize group sizes yields state-of-the-art worst-group-accuracy in the overparameterized regime. In this paper, we complement these empirical studies with a theoretical investigation of the risk of overparameterized random feature models on minority groups. In a setting in which the regression functions for the majority and minority groups are different, we show that overparameterization always improves minority group performance.

Related papers

• The Group Robustness is in the Details: Revisiting Finetuning under Spurious Correlations [8.844894807922902]
  Modern machine learning models are prone to over-reliance on spurious correlations. In this paper, we identify surprising and nuanced behavior of finetuned models on worst-group accuracy. Our results show more nuanced interactions of modern finetuned models with group robustness than was previously known.
  arXiv  Detail & Related papers  (2024-07-19T00:34:03Z)
• How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance [64.1656365676171]
  Group imbalance has been a known problem in empirical risk minimization. This paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance.
  arXiv  Detail & Related papers  (2024-03-12T04:38:05Z)
• Adversarial Reweighting Guided by Wasserstein Distance for Bias Mitigation [24.160692009892088]
  Under-representation of minorities in the data makes the disparate treatment of subpopulations difficult to deal with during learning. We propose a novel adversarial reweighting method to address such emphrepresentation bias.
  arXiv  Detail & Related papers  (2023-11-21T15:46:11Z)
• Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization [61.39201891894024]
  Group distributionally robust optimization (group DRO) can minimize the worst-case loss over pre-defined groups. We reformulate the group DRO framework by proposing Q-Diversity. Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.
  arXiv  Detail & Related papers  (2023-05-20T07:02:27Z)
• Ranking & Reweighting Improves Group Distributional Robustness [14.021069321266516]
  We propose a ranking-based training method called Discounted Rank Upweighting (DRU) to learn models that exhibit strong OOD performance on the test data. Results on several synthetic and real-world datasets highlight the superior ability of our group-ranking-based (akin to soft-minimax) approach in selecting and learning models that are robust to group distributional shifts.
  arXiv  Detail & Related papers  (2023-05-09T20:37:16Z)
• Outlier-Robust Group Inference via Gradient Space Clustering [50.87474101594732]
  Existing methods can improve the worst-group performance, but they require group annotations, which are often expensive and sometimes infeasible to obtain. We address the problem of learning group annotations in the presence of outliers by clustering the data in the space of gradients of the model parameters. We show that data in the gradient space has a simpler structure while preserving information about minority groups and outliers, making it suitable for standard clustering methods like DBSCAN.
  arXiv  Detail & Related papers  (2022-10-13T06:04:43Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations [98.3066727301239]
  We identify two key properties of the training data that drive this behavior. We show how the inductive bias of models towards "memorizing" fewer examples can cause over parameterization to hurt.
  arXiv  Detail & Related papers  (2020-05-09T01:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.