Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data

• URL: http://arxiv.org/abs/2103.03399v1
• Date: Fri, 5 Mar 2021 00:27:08 GMT
• Title: Representation Matters: Assessing the Importance of Subgroup Allocations in Training Data
• Authors: Esther Rolf, Theodora Worledge, Benjamin Recht, and Michael I. Jordan
• Abstract summary: We show that diverse representation in training data is key to increasing subgroup performances and achieving population level objectives. Our analysis and experiments describe how dataset compositions influence performance and provide constructive results for using trends in existing data, alongside domain knowledge, to help guide intentional, objective-aware dataset design.
• Score: 85.43008636875345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Collecting more diverse and representative training data is often touted as a remedy for the disparate performance of machine learning predictors across subpopulations. However, a precise framework for understanding how dataset properties like diversity affect learning outcomes is largely lacking. By casting data collection as part of the learning process, we demonstrate that diverse representation in training data is key not only to increasing subgroup performances, but also to achieving population level objectives. Our analysis and experiments describe how dataset compositions influence performance and provide constructive results for using trends in existing data, alongside domain knowledge, to help guide intentional, objective-aware dataset design.

Related papers

• Word Matters: What Influences Domain Adaptation in Summarization? [43.7010491942323]
  This paper investigates the fine-grained factors affecting domain adaptation performance. We propose quantifying dataset learning difficulty as the learning difficulty of generative summarization. Our experiments conclude that, when considering dataset learning difficulty, the cross-domain overlap and the performance gain in summarization tasks exhibit an approximate linear relationship.
  arXiv  Detail & Related papers  (2024-06-21T02:15:49Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Evaluating and Incentivizing Diverse Data Contributions in Collaborative Learning [89.21177894013225]
  For a federated learning model to perform well, it is crucial to have a diverse and representative dataset. We show that the statistical criterion used to quantify the diversity of the data, as well as the choice of the federated learning algorithm used, has a significant effect on the resulting equilibrium. We leverage this to design simple optimal federated learning mechanisms that encourage data collectors to contribute data representative of the global population.
  arXiv  Detail & Related papers  (2023-06-08T23:38:25Z)
• On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training [72.8087629914444]
  We study the impact of the trade-off between the intra-class diversity (the number of samples per class) and the inter-class diversity (the number of classes) of a supervised pre-training dataset. With the size of the pre-training dataset fixed, the best downstream performance comes with a balance on the intra-/inter-class diversity.
  arXiv  Detail & Related papers  (2023-05-20T16:23:50Z)
• Striving for data-model efficiency: Identifying data externalities on group performance [75.17591306911015]
  Building trustworthy, effective, and responsible machine learning systems hinges on understanding how differences in training data and modeling decisions interact to impact predictive performance. We focus on a particular type of data-model inefficiency, in which adding training data from some sources can actually lower performance evaluated on key sub-groups of the population. Our results indicate that data-efficiency is a key component of both accurate and trustworthy machine learning.
  arXiv  Detail & Related papers  (2022-11-11T16:48:27Z)
• Adaptive Sampling Strategies to Construct Equitable Training Datasets [0.7036032466145111]
  In domains ranging from computer vision to natural language processing, machine learning models have been shown to exhibit stark disparities. One factor contributing to these performance gaps is a lack of representation in the data the models are trained on. We formalize the problem of creating equitable training datasets, and propose a statistical framework for addressing this problem.
  arXiv  Detail & Related papers  (2022-01-31T19:19:30Z)
• Data-Centric Machine Learning in the Legal Domain [0.2624902795082451]
  This paper explores how changes in a data set influence the measured performance of a model. Using three publicly available data sets from the legal domain, we investigate how changes to their size, the train/test splits, and the human labelling accuracy impact the performance. The observed effects are surprisingly pronounced, especially when the per-class performance is considered.
  arXiv  Detail & Related papers  (2022-01-17T23:05:14Z)
• Towards Group Robustness in the presence of Partial Group Labels [61.33713547766866]
  spurious correlations between input samples and the target labels wrongly direct the neural network predictions. We propose an algorithm that optimize for the worst-off group assignments from a constraint set. We show improvements in the minority group's performance while preserving overall aggregate accuracy across groups.
  arXiv  Detail & Related papers  (2022-01-10T22:04:48Z)
• Sequential Targeting: an incremental learning approach for data imbalance in text classification [7.455546102930911]
  Methods to handle imbalanced datasets are crucial for alleviating distributional skews. We propose a novel training method, Sequential Targeting(ST), independent of the effectiveness of the representation method. We demonstrate the effectiveness of our method through experiments on simulated benchmark datasets (IMDB) and data collected from NAVER.
  arXiv  Detail & Related papers  (2020-11-20T04:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.