How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis

• URL: http://arxiv.org/abs/2406.09745v1
• Date: Fri, 14 Jun 2024 06:28:17 GMT
• Title: How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis
• Authors: Yuxin Dong, Tieliang Gong, Hong Chen, Shuangyong Song, Weizhan Zhang, Chen Li,
• Abstract summary: We formulate domain generalization from a novel probabilistic perspective. We provide key insights into the roles of gradient and representation matching in promoting generalization. In light of these theoretical findings, we introduce IDM to simultaneously align the inter-domain gradients and representations.
• Score: 21.685468628033206
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Domain generalization aims to learn invariance across multiple training domains, thereby enhancing generalization against out-of-distribution data. While gradient or representation matching algorithms have achieved remarkable success, these methods generally lack generalization guarantees or depend on strong assumptions, leaving a gap in understanding the underlying mechanism of distribution matching. In this work, we formulate domain generalization from a novel probabilistic perspective, ensuring robustness while avoiding overly conservative solutions. Through comprehensive information-theoretic analysis, we provide key insights into the roles of gradient and representation matching in promoting generalization. Our results reveal the complementary relationship between these two components, indicating that existing works focusing solely on either gradient or representation alignment are insufficient to solve the domain generalization problem. In light of these theoretical findings, we introduce IDM to simultaneously align the inter-domain gradients and representations. Integrated with the proposed PDM method for complex distribution matching, IDM achieves superior performance over various baseline methods.

Related papers

• Causally Inspired Regularization Enables Domain General Representations [14.036422506623383]
  Given a causal graph representing the data-generating process shared across different domains/distributions, enforcing sufficient graph-implied conditional independencies can identify domain-general (non-spurious) feature representations. We propose a novel framework with regularizations, which we demonstrate are sufficient for identifying domain-general feature representations without a priori knowledge (or proxies) of the spurious features. Our proposed method is effective for both (semi) synthetic and real-world data, outperforming other state-of-the-art methods in average and worst-domain transfer accuracy.
  arXiv  Detail & Related papers  (2024-04-25T01:33:55Z)
• Domain Generalisation via Risk Distribution Matching [17.334794920092577]
  We propose a novel approach for domain generalisation (DG) leveraging risk distributions to characterise domains. In testing, we may observe similar, or potentially intensifying in magnitude, divergences between risk distributions. We show that Risk Distribution Matching (RDM) shows superior generalisation capability over state-of-the-art DG methods.
  arXiv  Detail & Related papers  (2023-10-28T05:23:55Z)
• Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization [48.90173060487124]
  Domain generalization (DG) enables a learning machine from multiple seen source domains to an unseen target one. mire consists of two key components, namely, Category-aware Data Mixing (CDM) and Adaptive Semantic Topology Refinement (ASTR) experiments on multiple DG benchmarks validate the effectiveness and robustness of the proposed mire.
  arXiv  Detail & Related papers  (2022-10-14T06:52:34Z)
• Compound Domain Generalization via Meta-Knowledge Encoding [55.22920476224671]
  We introduce Style-induced Domain-specific Normalization (SDNorm) to re-normalize the multi-modal underlying distributions. We harness the prototype representations, the centroids of classes, to perform relational modeling in the embedding space. Experiments on four standard Domain Generalization benchmarks reveal that COMEN exceeds the state-of-the-art performance without the need of domain supervision.
  arXiv  Detail & Related papers  (2022-03-24T11:54:59Z)
• Towards Principled Disentanglement for Domain Generalization [90.9891372499545]
  A fundamental challenge for machine learning models is generalizing to out-of-distribution (OOD) data. We first formalize the OOD generalization problem as constrained optimization, called Disentanglement-constrained Domain Generalization (DDG) Based on the transformation, we propose a primal-dual algorithm for joint representation disentanglement and domain generalization.
  arXiv  Detail & Related papers  (2021-11-27T07:36:32Z)
• Self-balanced Learning For Domain Generalization [64.99791119112503]
  Domain generalization aims to learn a prediction model on multi-domain source data such that the model can generalize to a target domain with unknown statistics. Most existing approaches have been developed under the assumption that the source data is well-balanced in terms of both domain and class. We propose a self-balanced domain generalization framework that adaptively learns the weights of losses to alleviate the bias caused by different distributions of the multi-domain source data.
  arXiv  Detail & Related papers  (2021-08-31T03:17:54Z)
• Dual Reweighting Domain Generalization for Face Presentation Attack Detection [40.63170532438904]
  Face anti-spoofing approaches based on domain generalization (DG) have drawn growing attention due to their robustness for unseen scenarios. Previous methods treat each sample from multiple domains indiscriminately during the training process. We propose a novel Dual Reweighting Domain Generalization framework which iteratively reweights the relative importance between samples to further improve the generalization.
  arXiv  Detail & Related papers  (2021-06-30T15:24:34Z)
• An Online Learning Approach to Interpolation and Extrapolation in Domain Generalization [53.592597682854944]
  We recast generalization over sub-groups as an online game between a player minimizing risk and an adversary presenting new test. We show that ERM is provably minimax-optimal for both tasks.
  arXiv  Detail & Related papers  (2021-02-25T19:06:48Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.