In Search of Forgotten Domain Generalization

• URL: http://arxiv.org/abs/2410.08258v1
• Date: Thu, 10 Oct 2024 17:50:45 GMT
• Title: In Search of Forgotten Domain Generalization
• Authors: Prasanna Mayilvahanan, Roland S. Zimmermann, Thaddäus Wiedemer, Evgenia Rusak, Attila Juhos, Matthias Bethge, Wieland Brendel,
• Abstract summary: Out-of-Domain (OOD) generalization is the ability of a model trained on one or more domains to generalize to unseen domains. In the ImageNet era of computer vision, evaluation sets for measuring a model's OOD performance were designed to be strictly OOD with respect to style. The emergence of foundation models and expansive web-scale datasets has obfuscated this evaluation process.
• Score: 20.26519807919284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Out-of-Domain (OOD) generalization is the ability of a model trained on one or more domains to generalize to unseen domains. In the ImageNet era of computer vision, evaluation sets for measuring a model's OOD performance were designed to be strictly OOD with respect to style. However, the emergence of foundation models and expansive web-scale datasets has obfuscated this evaluation process, as datasets cover a broad range of domains and risk test domain contamination. In search of the forgotten domain generalization, we create large-scale datasets subsampled from LAION -- LAION-Natural and LAION-Rendition -- that are strictly OOD to corresponding ImageNet and DomainNet test sets in terms of style. Training CLIP models on these datasets reveals that a significant portion of their performance is explained by in-domain examples. This indicates that the OOD generalization challenges from the ImageNet era still prevail and that training on web-scale data merely creates the illusion of OOD generalization. Furthermore, through a systematic exploration of combining natural and rendition datasets in varying proportions, we identify optimal mixing ratios for model generalization across these domains. Our datasets and results re-enable meaningful assessment of OOD robustness at scale -- a crucial prerequisite for improving model robustness.

Related papers

• SOOD-ImageNet: a Large-Scale Dataset for Semantic Out-Of-Distribution Image Classification and Semantic Segmentation [6.21476985578569]
  Out-of-Distribution (OOD) detection in computer vision is a crucial research area. SOOD-ImageNet is a novel dataset comprising around 1.6M images across 56 classes. It is designed for common computer vision tasks such as image classification and semantic segmentation under OOD conditions.
  arXiv  Detail & Related papers  (2024-09-02T09:37:39Z)
• Feature Protection For Out-of-distribution Generalization [24.072876186625855]
  We show that protecting pre-trained features leads to a fine-tuned model more robust to generalization. We show that protecting pre-trained features leads to a fine-tuned model more robust to OOD generalization.
  arXiv  Detail & Related papers  (2024-05-25T03:00:06Z)
• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [69.28268603137546]
  We show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. This synthetic data is employed to evaluate the robustness of pretrained segmenters. We demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters.
  arXiv  Detail & Related papers  (2023-12-14T18:56:07Z)
• GEO-Bench: Toward Foundation Models for Earth Monitoring [139.77907168809085]
  We propose a benchmark comprised of six classification and six segmentation tasks. This benchmark will be a driver of progress across a variety of Earth monitoring tasks.
  arXiv  Detail & Related papers  (2023-06-06T16:16:05Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models [58.78043959556283]
  We study the behaviors of models under different fine-tuning methodologies, including full fine-tuning and Low-Rank Adaptation (LoRA) fine-tuning methods. Our analysis uncovers that LoRA fine-tuning consistently exhibits significantly better OOD generalization performance than full fine-tuning across various scenarios.
  arXiv  Detail & Related papers  (2022-10-10T16:07:24Z)
• OOD-Probe: A Neural Interpretation of Out-of-Domain Generalization [18.129450295108423]
  We propose a flexible framework that evaluates OOD systems with finer granularity using a probing module. We find that representations always encode some information about the domain. The high probing results correlate to the domain generalization performances, leading to further directions in developing OOD generalization systems.
  arXiv  Detail & Related papers  (2022-08-25T21:58:01Z)
• Benchmarking the Robustness of Instance Segmentation Models [3.1287804585804073]
  This paper presents a comprehensive evaluation of instance segmentation models with respect to real-world image corruptions and out-of-domain image collections. The out-of-domain image evaluation shows the generalization capability of models, an essential aspect of real-world applications. Specifically, this benchmark study includes state-of-the-art network architectures, network backbones, normalization layers, models trained starting from scratch or ImageNet pretrained networks.
  arXiv  Detail & Related papers  (2021-09-02T17:50:07Z)
• Context-Conditional Adaptation for Recognizing Unseen Classes in Unseen Domains [48.17225008334873]
  We propose a feature generative framework integrated with a COntext COnditional Adaptive (COCOA) Batch-Normalization. The generated visual features better capture the underlying data distribution enabling us to generalize to unseen classes and domains at test-time. We thoroughly evaluate and analyse our approach on established large-scale benchmark - DomainNet.
  arXiv  Detail & Related papers  (2021-07-15T17:51:16Z)
• Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU [70.44344060176952]
  Intent classification is a major task in spoken language understanding (SLU) Recent works have shown that using extra data and labels can improve the OOD detection performance. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection.
  arXiv  Detail & Related papers  (2021-06-28T08:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.