ERM++: An Improved Baseline for Domain Generalization

• URL: http://arxiv.org/abs/2304.01973v3
• Date: Tue, 26 Mar 2024 22:46:10 GMT
• Title: ERM++: An Improved Baseline for Domain Generalization
• Authors: Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis, Kate Saenko, Bryan A. Plummer,
• Abstract summary: We show that Empirical Risk Minimization (ERM) can outperform most existing Domain Generalization (DG) methods. ERM has achieved such strong results while only tuning hyper- parameters such as learning rate, weight decay, batch size, and dropout. We call the resulting stronger baseline ERM++.
• Score: 69.80606575323691
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Domain Generalization (DG) measures a classifier's ability to generalize to new distributions of data it was not trained on. Recent work has shown that a hyperparameter-tuned Empirical Risk Minimization (ERM) training procedure, that is simply minimizing the empirical risk on the source domains, can outperform most existing DG methods. ERM has achieved such strong results while only tuning hyper-parameters such as learning rate, weight decay, batch size, and dropout. However there are additional hyperparameters which further limit overfitting and catastrophic forgetting. We therefore focus on tuning previously untuned hyper-parameters, including training amount, initialization, and additional regularizers. We call the resulting stronger baseline ERM++. ERM++ improves the performance of DG by over 5% compared to prior ERM baselines on a standard benchmark of 5 datasets with a ResNet-50 and over 15% with a ViT-B/16, and outperforms all SOTA methods on DomainBed with both architectures. We also explore the relationship between DG performance and similarity to pre-training data, and find that similarity to pre-training data distributions is an important driver of performance, but that ERM++ with stronger initializations can deliver strong performance even on dissimilar datasets.Code is released at https://github.com/piotr-teterwak/erm_plusplus.

Related papers

• Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization [28.977757627384165]
  Domain Domain (DG) aims to avoid the performance degradation of the model when the distribution shift between the limited training data and unseen test data occurs. Recently, foundation models with enormous parameters have been pre-trained with huge datasets, demonstrating strong generalization ability. Our framework achieves SOTA performance on five DG benchmarks, while only requiring training a small number of parameters without adding additional testing cost.
  arXiv  Detail & Related papers  (2024-07-21T07:50:49Z)
• TAIA: Large Language Models are Out-of-Distribution Data Learners [30.57872423927015]
  We propose an effective inference-time intervention method: Training All parameters but Inferring with only Attention (trainallInfAttn) trainallInfAttn achieves superior improvements compared to both the fully fine-tuned model and the base model in most scenarios. The high tolerance of trainallInfAttn to data mismatches makes it resistant to jailbreaking tuning and enhances specialized tasks using general data.
  arXiv  Detail & Related papers  (2024-05-30T15:57:19Z)
• PUMA: margin-based data pruning [51.12154122266251]
  We focus on data pruning, where some training samples are removed based on the distance to the model classification boundary (i.e., margin) We propose PUMA, a new data pruning strategy that computes the margin using DeepFool. We show that PUMA can be used on top of the current state-of-the-art methodology in robustness, and it is able to significantly improve the model performance unlike the existing data pruning strategies.
  arXiv  Detail & Related papers  (2024-05-10T08:02:20Z)
• Cross Domain Generative Augmentation: Domain Generalization with Latent Diffusion Models [11.309433257851122]
  Cross Domain Generative Augmentation (CDGA) generates synthetic images to fill the gap between all domains. We show that CDGA outperforms SOTA DG methods under the Domainbed benchmark.
  arXiv  Detail & Related papers  (2023-12-08T21:52:00Z)
• Adversarial Style Augmentation for Domain Generalization [41.72506801753435]
  We introduce a novel Adrial Style Augmentation (ASA) method, which explores broader style spaces by generating more effective statistics perturbation. To facilitate the application of ASA, we design a simple yet effective module, namely AdvStyle, which instantiates the ASA method in a plug-and-play manner. Our method significantly outperforms its competitors on the PACS dataset under the single source generalization setting.
  arXiv  Detail & Related papers  (2023-01-30T03:52:16Z)
• On Certifying and Improving Generalization to Unseen Domains [87.00662852876177]
  Domain Generalization aims to learn models whose performance remains high on unseen domains encountered at test-time. It is challenging to evaluate DG algorithms comprehensively using a few benchmark datasets. We propose a universal certification framework that can efficiently certify the worst-case performance of any DG method.
  arXiv  Detail & Related papers  (2022-06-24T16:29:43Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)
• A Batch Normalization Classifier for Domain Adaptation [0.0]
  Adapting a model to perform well on unforeseen data outside its training set is a common problem that continues to motivate new approaches. We demonstrate that application of batch normalization in the output layer, prior to softmax activation, results in improved generalization across visual data domains in a refined ResNet model.
  arXiv  Detail & Related papers  (2021-03-22T08:03:44Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.