Invariant Information Bottleneck for Domain Generalization
- URL: http://arxiv.org/abs/2106.06333v2
- Date: Mon, 14 Jun 2021 17:31:00 GMT
- Title: Invariant Information Bottleneck for Domain Generalization
- Authors: Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Colorado J. Reed, Jun
Zhang, Dongsheng Li, Kurt Keutzer, Han Zhao
- Abstract summary: We propose a novel algorithm that learns a minimally sufficient representation that is invariant across training and testing domains.
By minimizing the mutual information between the representation and inputs, IIB alleviates its reliance on pseudo-invariant features.
The results show that IIB outperforms invariant learning baseline (e.g. IRM) by an average of 2.8% and 3.8% accuracy over two evaluation metrics.
- Score: 39.62337297660974
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The main challenge for domain generalization (DG) is to overcome the
potential distributional shift between multiple training domains and unseen
test domains. One popular class of DG algorithms aims to learn representations
that have an invariant causal relation across the training domains. However,
certain features, called \emph{pseudo-invariant features}, may be invariant in
the training domain but not the test domain and can substantially decreases the
performance of existing algorithms. To address this issue, we propose a novel
algorithm, called Invariant Information Bottleneck (IIB), that learns a
minimally sufficient representation that is invariant across training and
testing domains. By minimizing the mutual information between the
representation and inputs, IIB alleviates its reliance on pseudo-invariant
features, which is desirable for DG. To verify the effectiveness of the IIB
principle, we conduct extensive experiments on large-scale DG benchmarks. The
results show that IIB outperforms invariant learning baseline (e.g. IRM) by an
average of 2.8\% and 3.8\% accuracy over two evaluation metrics.
Related papers
- Disentangling Masked Autoencoders for Unsupervised Domain Generalization [57.56744870106124]
Unsupervised domain generalization is fast gaining attention but is still far from well-studied.
Disentangled Masked Auto (DisMAE) aims to discover the disentangled representations that faithfully reveal intrinsic features.
DisMAE co-trains the asymmetric dual-branch architecture with semantic and lightweight variation encoders.
arXiv Detail & Related papers (2024-07-10T11:11:36Z) - Learning to Learn Domain-invariant Parameters for Domain Generalization [29.821634033299855]
Domain generalization (DG) aims to overcome this issue by capturing domain-invariant representations from source domains.
We propose two modules of Domain Decoupling and Combination (DDC) and Domain-invariance-guided Backpropagation (DIGB)
Our proposed method has achieved state-of-the-art performance with strong generalization capability.
arXiv Detail & Related papers (2022-11-04T07:19:34Z) - Adaptive Domain Generalization via Online Disagreement Minimization [17.215683606365445]
Domain Generalization aims to safely transfer a model to unseen target domains.
AdaODM adaptively modifies the source model at test time for different target domains.
Results show AdaODM stably improves the generalization capacity on unseen domains.
arXiv Detail & Related papers (2022-08-03T11:51:11Z) - Failure Modes of Domain Generalization Algorithms [29.772370301145543]
We propose an evaluation framework for domain generalization algorithms.
We show that the largest contributor to the generalization error varies across methods, datasets, regularization strengths and even training lengths.
arXiv Detail & Related papers (2021-11-26T20:04:19Z) - A Bit More Bayesian: Domain-Invariant Learning with Uncertainty [111.22588110362705]
Domain generalization is challenging due to the domain shift and the uncertainty caused by the inaccessibility of target domain data.
In this paper, we address both challenges with a probabilistic framework based on variational Bayesian inference.
We derive domain-invariant representations and classifiers, which are jointly established in a two-layer Bayesian neural network.
arXiv Detail & Related papers (2021-05-09T21:33:27Z) - Learning Invariant Representations and Risks for Semi-supervised Domain
Adaptation [109.73983088432364]
We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA)
We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
arXiv Detail & Related papers (2020-10-09T15:42:35Z) - Adaptively-Accumulated Knowledge Transfer for Partial Domain Adaptation [66.74638960925854]
Partial domain adaptation (PDA) deals with a realistic and challenging problem when the source domain label space substitutes the target domain.
We propose an Adaptively-Accumulated Knowledge Transfer framework (A$2$KT) to align the relevant categories across two domains.
arXiv Detail & Related papers (2020-08-27T00:53:43Z) - Dual Mixup Regularized Learning for Adversarial Domain Adaptation [19.393393465837377]
We propose a dual mixup regularized learning (DMRL) method for unsupervised domain adaptation.
DMRL guides the classifier in enhancing consistent predictions in-between samples, and enriches the intrinsic structures of the latent space.
A series of empirical studies on four domain adaptation benchmarks demonstrate that our approach can achieve the state-of-the-art.
arXiv Detail & Related papers (2020-07-07T00:24:14Z) - Gradually Vanishing Bridge for Adversarial Domain Adaptation [156.46378041408192]
We equip adversarial domain adaptation with Gradually Vanishing Bridge (GVB) mechanism on both generator and discriminator.
On the generator, GVB could not only reduce the overall transfer difficulty, but also reduce the influence of the residual domain-specific characteristics.
On the discriminator, GVB contributes to enhance the discriminating ability, and balance the adversarial training process.
arXiv Detail & Related papers (2020-03-30T01:36:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.