ERM++: An Improved Baseline for Domain Generalization

• URL: http://arxiv.org/abs/2304.01973v3
• Date: Tue, 26 Mar 2024 22:46:10 GMT
• Title: ERM++: An Improved Baseline for Domain Generalization
• Authors: Piotr Teterwak, Kuniaki Saito, Theodoros Tsiligkaridis, Kate Saenko, Bryan A. Plummer,
• Abstract summary: We show that Empirical Risk Minimization (ERM) can outperform most existing Domain Generalization (DG) methods. ERM has achieved such strong results while only tuning hyper- parameters such as learning rate, weight decay, batch size, and dropout. We call the resulting stronger baseline ERM++.
• Score: 69.80606575323691
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Domain Generalization (DG) measures a classifier's ability to generalize to new distributions of data it was not trained on. Recent work has shown that a hyperparameter-tuned Empirical Risk Minimization (ERM) training procedure, that is simply minimizing the empirical risk on the source domains, can outperform most existing DG methods. ERM has achieved such strong results while only tuning hyper-parameters such as learning rate, weight decay, batch size, and dropout. However there are additional hyperparameters which further limit overfitting and catastrophic forgetting. We therefore focus on tuning previously untuned hyper-parameters, including training amount, initialization, and additional regularizers. We call the resulting stronger baseline ERM++. ERM++ improves the performance of DG by over 5% compared to prior ERM baselines on a standard benchmark of 5 datasets with a ResNet-50 and over 15% with a ViT-B/16, and outperforms all SOTA methods on DomainBed with both architectures. We also explore the relationship between DG performance and similarity to pre-training data, and find that similarity to pre-training data distributions is an important driver of performance, but that ERM++ with stronger initializations can deliver strong performance even on dissimilar datasets.Code is released at https://github.com/piotr-teterwak/erm_plusplus.

Related papers

• Is Large-Scale Pretraining the Secret to Good Domain Generalization? [69.80606575323691]
  Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains. Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results. We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP.
  arXiv  Detail & Related papers  (2024-12-03T21:43:11Z)
• Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization [28.977757627384165]
  Domain Domain (DG) aims to avoid the performance degradation of the model when the distribution shift between the limited training data and unseen test data occurs. Recently, foundation models with enormous parameters have been pre-trained with huge datasets, demonstrating strong generalization ability. Our framework achieves SOTA performance on five DG benchmarks, while only requiring training a small number of parameters without adding additional testing cost.
  arXiv  Detail & Related papers  (2024-07-21T07:50:49Z)
• TAIA: Large Language Models are Out-of-Distribution Data Learners [30.57872423927015]
  We propose an effective inference-time intervention method: Training All parameters but Inferring with only Attention (trainallInfAttn) trainallInfAttn achieves superior improvements compared to both the fully fine-tuned model and the base model in most scenarios. The high tolerance of trainallInfAttn to data mismatches makes it resistant to jailbreaking tuning and enhances specialized tasks using general data.
  arXiv  Detail & Related papers  (2024-05-30T15:57:19Z)
• PUMA: margin-based data pruning [51.12154122266251]
  We focus on data pruning, where some training samples are removed based on the distance to the model classification boundary (i.e., margin) We propose PUMA, a new data pruning strategy that computes the margin using DeepFool. We show that PUMA can be used on top of the current state-of-the-art methodology in robustness, and it is able to significantly improve the model performance unlike the existing data pruning strategies.
  arXiv  Detail & Related papers  (2024-05-10T08:02:20Z)
• Efficiently Assemble Normalization Layers and Regularization for Federated Domain Generalization [1.1534313664323637]
  Domain shift is a formidable issue in Machine Learning that causes a model to suffer from performance degradation when tested on unseen domains. FedDG attempts to train a global model using collaborative clients in a privacy-preserving manner that can generalize well to unseen clients possibly with domain shift. Here, we introduce a novel architectural method for FedDG, namely gPerXAN, which relies on a normalization scheme working with a guiding regularizer.
  arXiv  Detail & Related papers  (2024-03-22T20:22:08Z)
• Cross Domain Generative Augmentation: Domain Generalization with Latent Diffusion Models [11.309433257851122]
  Cross Domain Generative Augmentation (CDGA) generates synthetic images to fill the gap between all domains. We show that CDGA outperforms SOTA DG methods under the Domainbed benchmark.
  arXiv  Detail & Related papers  (2023-12-08T21:52:00Z)
• Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders [63.28408887247742]
  We study whether training procedures can be improved to yield better generalization capabilities in the resulting models. We recommend a simple recipe for training dense encoders: Train on MSMARCO with parameter-efficient methods, such as LoRA, and opt for using in-batch negatives unless given well-constructed hard negatives.
  arXiv  Detail & Related papers  (2023-11-16T10:42:58Z)
• Adversarial Style Augmentation for Domain Generalization [41.72506801753435]
  We introduce a novel Adrial Style Augmentation (ASA) method, which explores broader style spaces by generating more effective statistics perturbation. To facilitate the application of ASA, we design a simple yet effective module, namely AdvStyle, which instantiates the ASA method in a plug-and-play manner. Our method significantly outperforms its competitors on the PACS dataset under the single source generalization setting.
  arXiv  Detail & Related papers  (2023-01-30T03:52:16Z)
• On-Device Domain Generalization [93.79736882489982]
  Domain generalization is critical to on-device machine learning applications. We find that knowledge distillation is a strong candidate for solving the problem. We propose a simple idea called out-of-distribution knowledge distillation (OKD), which aims to teach the student how the teacher handles (synthetic) out-of-distribution data.
  arXiv  Detail & Related papers  (2022-09-15T17:59:31Z)
• Back-to-Bones: Rediscovering the Role of Backbones in Domain Generalization [1.6799377888527687]
  Domain Generalization studies the capability of a deep learning model to generalize to out-of-training distributions. Recent research has provided a reproducible benchmark for DG, pointing out the effectiveness of naive empirical risk minimization (ERM) over existing algorithms. In this paper, we evaluate the backbones proposing a comprehensive analysis of their intrinsic generalization capabilities.
  arXiv  Detail & Related papers  (2022-09-02T15:30:17Z)
• On Certifying and Improving Generalization to Unseen Domains [87.00662852876177]
  Domain Generalization aims to learn models whose performance remains high on unseen domains encountered at test-time. It is challenging to evaluate DG algorithms comprehensively using a few benchmark datasets. We propose a universal certification framework that can efficiently certify the worst-case performance of any DG method.
  arXiv  Detail & Related papers  (2022-06-24T16:29:43Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Improving Multi-Domain Generalization through Domain Re-labeling [31.636953426159224]
  We study the important link between pre-specified domain labels and the generalization performance. We introduce a general approach for multi-domain generalization, MulDEns, that uses an ERM-based deep ensembling backbone. We show that MulDEns does not require tailoring the augmentation strategy or the training process specific to a dataset.
  arXiv  Detail & Related papers  (2021-12-17T23:21:50Z)
• META: Mimicking Embedding via oThers' Aggregation for Generalizable Person Re-identification [68.39849081353704]
  Domain generalizable (DG) person re-identification (ReID) aims to test across unseen domains without access to the target domain data at training time. This paper presents a new approach called Mimicking Embedding via oThers' Aggregation (META) for DG ReID.
  arXiv  Detail & Related papers  (2021-12-16T08:06:50Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)
• Reappraising Domain Generalization in Neural Networks [8.06370138649329]
  Domain generalization (DG) of machine learning algorithms is defined as their ability to learn a domain agnostic hypothesis from multiple training distributions. We find that a straightforward Empirical Risk Minimization (ERM) baseline consistently outperforms existing DG methods. We propose a classwise-DG formulation, where for each class, we randomly select one of the domains and keep it aside for testing.
  arXiv  Detail & Related papers  (2021-10-15T10:06:40Z)
• A Batch Normalization Classifier for Domain Adaptation [0.0]
  Adapting a model to perform well on unforeseen data outside its training set is a common problem that continues to motivate new approaches. We demonstrate that application of batch normalization in the output layer, prior to softmax activation, results in improved generalization across visual data domains in a refined ResNet model.
  arXiv  Detail & Related papers  (2021-03-22T08:03:44Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.