On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalization

• URL: http://arxiv.org/abs/2212.09082v1
• Date: Sun, 18 Dec 2022 13:13:44 GMT
• Title: On the Connection between Invariant Learning and Adversarial Training for Out-of-Distribution Generalization
• Authors: Shiji Xin, Yifei Wang, Jingtong Su, Yisen Wang
• Abstract summary: deep learning models rely on spurious features, which catastrophically fail when generalized to out-of-distribution (OOD) data. Recent work shows that Invariant Risk Minimization (IRM) is only effective for a certain type of distribution shift while it fails for other cases. We propose Domainwise Adversarial Training ( DAT), an AT-inspired method for alleviating distribution shift by domain-specific perturbations.
• Score: 14.233038052654484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite impressive success in many tasks, deep learning models are shown to rely on spurious features, which will catastrophically fail when generalized to out-of-distribution (OOD) data. Invariant Risk Minimization (IRM) is proposed to alleviate this issue by extracting domain-invariant features for OOD generalization. Nevertheless, recent work shows that IRM is only effective for a certain type of distribution shift (e.g., correlation shift) while it fails for other cases (e.g., diversity shift). Meanwhile, another thread of method, Adversarial Training (AT), has shown better domain transfer performance, suggesting that it has the potential to be an effective candidate for extracting domain-invariant features. This paper investigates this possibility by exploring the similarity between the IRM and AT objectives. Inspired by this connection, we propose Domainwise Adversarial Training (DAT), an AT-inspired method for alleviating distribution shift by domain-specific perturbations. Extensive experiments show that our proposed DAT can effectively remove domain-varying features and improve OOD generalization under both correlation shift and diversity shift.

Related papers

• Mimicking Better by Matching the Approximate Action Distribution [48.95048003354255]
  We introduce MAAD, a novel, sample-efficient on-policy algorithm for Imitation Learning from Observations. We show that it requires considerable fewer interactions to achieve expert performance, outperforming current state-of-the-art on-policy methods.
  arXiv  Detail & Related papers  (2023-06-16T12:43:47Z)
• Source-free Domain Adaptation Requires Penalized Diversity [60.04618512479438]
  Source-free domain adaptation (SFDA) was introduced to address knowledge transfer between different domains in the absence of source data. In unsupervised SFDA, the diversity is limited to learning a single hypothesis on the source or learning multiple hypotheses with a shared feature extractor. We propose a novel unsupervised SFDA algorithm that promotes representational diversity through the use of separate feature extractors.
  arXiv  Detail & Related papers  (2023-04-06T00:20:19Z)
• MADAv2: Advanced Multi-Anchor Based Active Domain Adaptation Segmentation [98.09845149258972]
  We introduce active sample selection to assist domain adaptation regarding the semantic segmentation task. With only a little workload to manually annotate these samples, the distortion of the target-domain distribution can be effectively alleviated. A powerful semi-supervised domain adaptation strategy is proposed to alleviate the long-tail distribution problem.
  arXiv  Detail & Related papers  (2023-01-18T07:55:22Z)
• Improving Transferability of Domain Adaptation Networks Through Domain Alignment Layers [1.3766148734487902]
  Multi-source unsupervised domain adaptation (MSDA) aims at learning a predictor for an unlabeled domain by assigning weak knowledge from a bag of source models. We propose to embed Multi-Source version of DomaIn Alignment Layers (MS-DIAL) at different levels of the predictor. Our approach can improve state-of-the-art MSDA methods, yielding relative gains of up to +30.64% on their classification accuracies.
  arXiv  Detail & Related papers  (2021-09-06T18:41:19Z)
• Dynamic Domain Adaptation for Efficient Inference [12.713628738434881]
  Domain adaptation (DA) enables knowledge transfer from a labeled source domain to an unlabeled target domain. Most prior DA approaches leverage complicated and powerful deep neural networks to improve the adaptation capacity. We propose a dynamic domain adaptation (DDA) framework, which can simultaneously achieve efficient target inference in low-resource scenarios.
  arXiv  Detail & Related papers  (2021-03-26T08:53:16Z)
• Margin Preserving Self-paced Contrastive Learning Towards Domain Adaptation for Medical Image Segmentation [51.93711960601973]
  We propose a novel margin preserving self-paced contrastive Learning model for cross-modal medical image segmentation. With the guidance of progressively refined semantic prototypes, a novel margin preserving contrastive loss is proposed to boost the discriminability of embedded representation space. Experiments on cross-modal cardiac segmentation tasks demonstrate that MPSCL significantly improves semantic segmentation performance.
  arXiv  Detail & Related papers  (2021-03-15T15:23:10Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• Adaptively-Accumulated Knowledge Transfer for Partial Domain Adaptation [66.74638960925854]
  Partial domain adaptation (PDA) deals with a realistic and challenging problem when the source domain label space substitutes the target domain. We propose an Adaptively-Accumulated Knowledge Transfer framework (A$2$KT) to align the relevant categories across two domains.
  arXiv  Detail & Related papers  (2020-08-27T00:53:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.