Towards Reliable Domain Generalization: A New Dataset and Evaluations
- URL: http://arxiv.org/abs/2309.06142v1
- Date: Tue, 12 Sep 2023 11:29:12 GMT
- Title: Towards Reliable Domain Generalization: A New Dataset and Evaluations
- Authors: Jiao Zhang, Xu-Yao Zhang, Cheng-Lin Liu
- Abstract summary: We propose a new domain generalization task for handwritten Chinese character recognition (HCCR)
We evaluate eighteen DG methods on the proposed PaHCC dataset and show that the performance of existing methods is still unsatisfactory.
Our dataset and evaluations bring new perspectives to the community for more substantial progress.
- Score: 45.68339440942477
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: There are ubiquitous distribution shifts in the real world. However, deep
neural networks (DNNs) are easily biased towards the training set, which causes
severe performance degradation when they receive out-of-distribution data. Many
methods are studied to train models that generalize under various distribution
shifts in the literature of domain generalization (DG). However, the recent
DomainBed and WILDS benchmarks challenged the effectiveness of these methods.
Aiming at the problems in the existing research, we propose a new domain
generalization task for handwritten Chinese character recognition (HCCR) to
enrich the application scenarios of DG method research. We evaluate eighteen DG
methods on the proposed PaHCC (Printed and Handwritten Chinese Characters)
dataset and show that the performance of existing methods on this dataset is
still unsatisfactory. Besides, under a designed dynamic DG setting, we reveal
more properties of DG methods and argue that only the leave-one-domain-out
protocol is unreliable. We advocate that researchers in the DG community refer
to dynamic performance of methods for more comprehensive and reliable
evaluation. Our dataset and evaluations bring new perspectives to the community
for more substantial progress. We will make our dataset public with the article
published to facilitate the study of domain generalization.
Related papers
- PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization [24.413415998529754]
We propose a new benchmark Hybrid Domain Generalization (HDG) and a novel metric $H2$-CV, which construct various splits to assess the robustness of algorithms.
Our method outperforms state-of-the-art algorithms on multiple datasets, especially improving the robustness when confronting data scarcity.
arXiv Detail & Related papers (2024-04-13T13:41:13Z) - Consistency Regularization for Generalizable Source-free Domain
Adaptation [62.654883736925456]
Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset.
Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets.
We propose a consistency regularization framework to develop a more generalizable SFDA method.
arXiv Detail & Related papers (2023-08-03T07:45:53Z) - Back-to-Bones: Rediscovering the Role of Backbones in Domain
Generalization [1.6799377888527687]
Domain Generalization studies the capability of a deep learning model to generalize to out-of-training distributions.
Recent research has provided a reproducible benchmark for DG, pointing out the effectiveness of naive empirical risk minimization (ERM) over existing algorithms.
In this paper, we evaluate the backbones proposing a comprehensive analysis of their intrinsic generalization capabilities.
arXiv Detail & Related papers (2022-09-02T15:30:17Z) - On Certifying and Improving Generalization to Unseen Domains [87.00662852876177]
Domain Generalization aims to learn models whose performance remains high on unseen domains encountered at test-time.
It is challenging to evaluate DG algorithms comprehensively using a few benchmark datasets.
We propose a universal certification framework that can efficiently certify the worst-case performance of any DG method.
arXiv Detail & Related papers (2022-06-24T16:29:43Z) - Reappraising Domain Generalization in Neural Networks [8.06370138649329]
Domain generalization (DG) of machine learning algorithms is defined as their ability to learn a domain agnostic hypothesis from multiple training distributions.
We find that a straightforward Empirical Risk Minimization (ERM) baseline consistently outperforms existing DG methods.
We propose a classwise-DG formulation, where for each class, we randomly select one of the domains and keep it aside for testing.
arXiv Detail & Related papers (2021-10-15T10:06:40Z) - COLUMBUS: Automated Discovery of New Multi-Level Features for Domain
Generalization via Knowledge Corruption [12.555885317622131]
We address the challenging domain generalization problem, where a model trained on a set of source domains is expected to generalize well in unseen domains without exposure to their data.
We propose Columbus, a method that enforces new feature discovery via a targeted corruption of the most relevant input and multi-level representations of the data.
arXiv Detail & Related papers (2021-09-09T14:52:05Z) - Semi-Supervised Domain Generalization with Stochastic StyleMatch [90.98288822165482]
In real-world applications, we might have only a few labels available from each source domain due to high annotation cost.
In this work, we investigate semi-supervised domain generalization, a more realistic and practical setting.
Our proposed approach, StyleMatch, is inspired by FixMatch, a state-of-the-art semi-supervised learning method based on pseudo-labeling.
arXiv Detail & Related papers (2021-06-01T16:00:08Z) - Domain Generalization: A Survey [146.68420112164577]
Domain generalization (DG) aims to achieve OOD generalization by only using source domain data for model learning.
For the first time, a comprehensive literature review is provided to summarize the ten-year development in DG.
arXiv Detail & Related papers (2021-03-03T16:12:22Z) - A Review of Single-Source Deep Unsupervised Visual Domain Adaptation [81.07994783143533]
Large-scale labeled training datasets have enabled deep neural networks to excel across a wide range of benchmark vision tasks.
In many applications, it is prohibitively expensive and time-consuming to obtain large quantities of labeled data.
To cope with limited labeled training data, many have attempted to directly apply models trained on a large-scale labeled source domain to another sparsely labeled or unlabeled target domain.
arXiv Detail & Related papers (2020-09-01T00:06:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.