Distributionally Robust Safe Sample Screening

• URL: http://arxiv.org/abs/2406.05964v1
• Date: Mon, 10 Jun 2024 01:46:42 GMT
• Title: Distributionally Robust Safe Sample Screening
• Authors: Hiroyuki Hanada, Aoyama Tatsuya, Akahane Satoshi, Tomonari Tanaka, Yoshito Okura, Yu Inatsu, Noriaki Hashimoto, Shion Takeno, Taro Murayama, Hanju Lee, Shinya Kojima, Ichiro Takeuchi,
• Abstract summary: We propose a machine learning method called Distributionally Robust Safe Sample Screening (DRSSS) DRSSS aims to identify unnecessary training samples, even when the distribution of the training samples changes in the future. We provide a theoretical guarantee for the DRSSS method and validate its performance through numerical experiments on both synthetic and real-world datasets.
• Score: 15.791952053731448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this study, we propose a machine learning method called Distributionally Robust Safe Sample Screening (DRSSS). DRSSS aims to identify unnecessary training samples, even when the distribution of the training samples changes in the future. To achieve this, we effectively combine the distributionally robust (DR) paradigm, which aims to enhance model robustness against variations in data distribution, with the safe sample screening (SSS), which identifies unnecessary training samples prior to model training. Since we need to consider an infinite number of scenarios regarding changes in the distribution, we applied SSS because it does not require model training after the change of the distribution. In this paper, we employed the covariate shift framework to represent the distribution of training samples and reformulated the DR covariate-shift problem as a weighted empirical risk minimization problem, where the weights are subject to uncertainty within a predetermined range. By extending the existing SSS technique to accommodate this weight uncertainty, the DRSSS method is capable of reliably identifying unnecessary samples under any future distribution within a specified range. We provide a theoretical guarantee for the DRSSS method and validate its performance through numerical experiments on both synthetic and real-world datasets.

Related papers

• Probabilistic Federated Prompt-Tuning with Non-IID and Imbalanced Data [35.47385526394076]
  Fine-tuning pre-trained models is a popular approach in machine learning for solving complex tasks with moderate data. Fine-tuning the entire pre-trained model is ineffective in federated data scenarios where local data distributions are diversely skewed. Our approach transforms federated learning into a distributed set modeling task, aggregating diverse sets of prompts to globally fine-tune the pre-trained model.
  arXiv  Detail & Related papers  (2025-02-27T04:31:34Z)
• Distributionally Robust Coreset Selection under Covariate Shift [14.979902937312099]
  Coreset selection is an approach to reducing training data. It is challenging to select an effective subset of training data that performs well across all deployment scenarios. We propose Distributionally Robust Coreset Selection (DRCS) for this purpose.
  arXiv  Detail & Related papers  (2025-01-24T05:36:53Z)
• Adaptive teachers for amortized samplers [76.88721198565861]
  We propose an adaptive training distribution (the teacher) to guide the training of the primary amortized sampler (the student)<n>We validate the effectiveness of this approach in a synthetic environment designed to present an exploration challenge.
  arXiv  Detail & Related papers  (2024-10-02T11:33:13Z)
• Dataset Quantization with Active Learning based Adaptive Sampling [11.157462442942775]
  We show that maintaining performance is feasible even with uneven sample distributions. We propose a novel active learning based adaptive sampling strategy to optimize the sample selection. Our approach outperforms the state-of-the-art dataset compression methods.
  arXiv  Detail & Related papers  (2024-07-09T23:09:18Z)
• Training Implicit Generative Models via an Invariant Statistical Loss [3.139474253994318]
  Implicit generative models have the capability to learn arbitrary complex data distributions. On the downside, training requires telling apart real data from artificially-generated ones using adversarial discriminators. We develop a discriminator-free method for training one-dimensional (1D) generative implicit models.
  arXiv  Detail & Related papers  (2024-02-26T09:32:28Z)
• Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains [9.429772474335122]
  We focus on scenarios where data distributions vary across multiple segments of the entire population. We propose a two-stage multiply robust estimation method to improve model performance on each individual segment. Our method is designed to be implemented with commonly used off-the-shelf machine learning models.
  arXiv  Detail & Related papers  (2024-02-21T22:01:10Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Multi-Domain Joint Training for Person Re-Identification [51.73921349603597]
  Deep learning-based person Re-IDentification (ReID) often requires a large amount of training data to achieve good performance. It appears that collecting more training data from diverse environments tends to improve the ReID performance. We propose an approach called Domain-Camera-Sample Dynamic network (DCSD) whose parameters can be adaptive to various factors.
  arXiv  Detail & Related papers  (2022-01-06T09:20:59Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• WILDS: A Benchmark of in-the-Wild Distribution Shifts [157.53410583509924]
  Distribution shifts can substantially degrade the accuracy of machine learning systems deployed in the wild. We present WILDS, a curated collection of 8 benchmark datasets that reflect a diverse range of distribution shifts. We show that standard training results in substantially lower out-of-distribution than in-distribution performance.
  arXiv  Detail & Related papers  (2020-12-14T11:14:56Z)
• Attentional-Biased Stochastic Gradient Descent [74.49926199036481]
  We present a provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning. Our method is a simple modification to momentum SGD where we assign an individual importance weight to each sample in the mini-batch. ABSGD is flexible enough to combine with other robust losses without any additional cost.
  arXiv  Detail & Related papers  (2020-12-13T03:41:52Z)
• Robust Federated Learning: The Case of Affine Distribution Shifts [41.27887358989414]
  We develop a robust federated learning algorithm that achieves satisfactory performance against distribution shifts in users' samples. We show that an affine distribution shift indeed suffices to significantly decrease the performance of the learnt classifier in a new test user.
  arXiv  Detail & Related papers  (2020-06-16T03:43:59Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)
• Reinforced Data Sampling for Model Diversification [15.547681142342846]
  This paper proposes a new Reinforced Data Sampling (RDS) method to learn how to sample data adequately. We formulate the optimisation problem of model diversification $delta-div$ in data sampling to maximise learning potentials and optimum allocation by injecting model diversity. Our results suggest that the trainable sampling for model diversification is useful for competition organisers, researchers, or even starters to pursue full potentials of various machine learning tasks.
  arXiv  Detail & Related papers  (2020-06-12T11:46:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.