Invariance Principle Meets Vicinal Risk Minimization

• URL: http://arxiv.org/abs/2407.05765v2
• Date: Thu, 23 Jan 2025 15:42:16 GMT
• Title: Invariance Principle Meets Vicinal Risk Minimization
• Authors: Yaoyao Zhu, Xiuding Cai, Yingkai Wang, Dong Miao, Zhongliang Fu, Xu Luo,
• Abstract summary: Invariant Risk Minimization (IRM) aims to address OOD generalization by learning domain-invariant features.<n>We propose a domain-shared Semantic Data Augmentation (SDA) module, designed to enhance dataset diversity while maintaining label consistency.
• Score: 2.026281591452464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning models excel in computer vision tasks but often fail to generalize to out-of-distribution (OOD) domains. Invariant Risk Minimization (IRM) aims to address OOD generalization by learning domain-invariant features. However, IRM struggles with datasets exhibiting significant diversity shifts. While data augmentation methods like Mixup and Semantic Data Augmentation (SDA) enhance diversity, they risk over-augmentation and label instability. To address these challenges, we propose a domain-shared Semantic Data Augmentation (SDA) module, a novel implementation of Variance Risk Minimization (VRM) designed to enhance dataset diversity while maintaining label consistency. We further provide a Rademacher complexity analysis, establishing a tighter generalization error bound compared to baseline methods. Extensive evaluations on OOD benchmarks, including PACS, VLCS, OfficeHome, and TerraIncognita, demonstrate consistent performance improvements over state-of-the-art domain generalization methods.

Related papers

• Domain Generalized Stereo Matching with Uncertainty-guided Data Augmentation [11.938635624781313]
  State-of-the-art stereo matching (SM) models often fail to generalize to real data domains due to domain differences.<n>We leverage data augmentation to expand the training domain, encouraging the model to acquire robust cross-domain feature representations.<n>Our approach is simple, architecture-agnostic, and can be integrated into any SM networks.
  arXiv  Detail & Related papers  (2025-08-02T10:26:53Z)
• Weight Averaging for Out-of-Distribution Generalization and Few-Shot Domain Adaptation [0.0]
  Two techniques have been developed for addressing out-of-distribution generalization in computer vision. We propose increasing the model diversity in WA explicitly by gradient similarity as a loss regularizer. We also propose combining WA and SAM to solve the problem of few-shot domain adaptation.
  arXiv  Detail & Related papers  (2025-01-14T10:04:05Z)
• Towards Robust Multimodal Sentiment Analysis with Incomplete Data [20.75292807497547]
  We present an innovative Language-dominated Noise-resistant Learning Network (LNLN) to achieve robust Multimodal Sentiment Analysis (MSA) LNLN features a dominant modality correction (DMC) module and dominant modality based multimodal learning (DMML) module, which enhances the model's robustness across various noise scenarios.
  arXiv  Detail & Related papers  (2024-09-30T07:14:31Z)
• DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series [61.91288852233078]
  In time series anomaly detection, the scarcity of labeled data poses a challenge to the development of accurate models.<n>We propose a novel Domain Contrastive learning model for Anomaly Detection in time series (DACAD)<n>Our model employs supervised contrastive loss for the source domain and self-supervised contrastive triplet loss for the target domain.
  arXiv  Detail & Related papers  (2024-04-17T11:20:14Z)
• Diagnosing and Rectifying Fake OOD Invariance: A Restructured Causal Approach [51.012396632595554]
  Invariant representation learning (IRL) encourages the prediction from invariant causal features to labels de-confounded from the environments. Recent theoretical results verified that some causal features recovered by IRLs merely pretend domain-invariantly in the training environments but fail in unseen domains. We develop an approach based on conditional mutual information with respect to RS-SCM, then rigorously rectify the spurious and fake invariant effects.
  arXiv  Detail & Related papers  (2023-12-15T12:58:05Z)
• Continual Invariant Risk Minimization [46.051656238770086]
  Empirical risk minimization can lead to poor generalization behavior on unseen environments if the learned model does not capture invariant feature representations. Invariant risk minimization (IRM) is a recent proposal for discovering environment-invariant representations.
  arXiv  Detail & Related papers  (2023-10-21T11:44:47Z)
• On the Variance, Admissibility, and Stability of Empirical Risk Minimization [80.26309576810844]
  Empirical Risk Minimization (ERM) with squared loss may attain minimax suboptimal error rates. We show that under mild assumptions, the suboptimality of ERM must be due to large bias rather than variance. We also show that our estimates imply stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for non-Donsker classes.
  arXiv  Detail & Related papers  (2023-05-29T15:25:48Z)
• Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization [61.39201891894024]
  Group distributionally robust optimization (group DRO) can minimize the worst-case loss over pre-defined groups. We reformulate the group DRO framework by proposing Q-Diversity. Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.
  arXiv  Detail & Related papers  (2023-05-20T07:02:27Z)
• Source-free Domain Adaptation Requires Penalized Diversity [60.04618512479438]
  Source-free domain adaptation (SFDA) was introduced to address knowledge transfer between different domains in the absence of source data. In unsupervised SFDA, the diversity is limited to learning a single hypothesis on the source or learning multiple hypotheses with a shared feature extractor. We propose a novel unsupervised SFDA algorithm that promotes representational diversity through the use of separate feature extractors.
  arXiv  Detail & Related papers  (2023-04-06T00:20:19Z)
• ERM++: An Improved Baseline for Domain Generalization [69.80606575323691]
  Empirical Risk Minimization (ERM) can outperform most more complex Domain Generalization (DG) methods when properly tuned. ERM++ improves DG performance by over 5% compared to prior ERM baselines.
  arXiv  Detail & Related papers  (2023-04-04T17:31:15Z)
• Learning Optimal Features via Partial Invariance [18.552839725370383]
  Invariant Risk Minimization (IRM) is a popular framework that aims to learn robust models from multiple environments. We show that IRM can over-constrain the predictor and to remedy this, we propose a relaxation via $textitpartial invariance$. Several experiments, conducted both in linear settings as well as with deep neural networks on tasks over both language and image data, allow us to verify our conclusions.
  arXiv  Detail & Related papers  (2023-01-28T02:48:14Z)
• Domain-Specific Risk Minimization for Out-of-Distribution Generalization [104.17683265084757]
  We first establish a generalization bound that explicitly considers the adaptivity gap. We propose effective gap estimation methods for guiding the selection of a better hypothesis for the target. The other method is minimizing the gap directly by adapting model parameters using online target samples.
  arXiv  Detail & Related papers  (2022-08-18T06:42:49Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Learning Invariant Representation with Consistency and Diversity for Semi-supervised Source Hypothesis Transfer [46.68586555288172]
  We propose a novel task named Semi-supervised Source Hypothesis Transfer (SSHT), which performs domain adaptation based on source trained model, to generalize well in target domain with a few supervisions. We propose Consistency and Diversity Learning (CDL), a simple but effective framework for SSHT by facilitating prediction consistency between two randomly augmented unlabeled data. Experimental results show that our method outperforms existing SSDA methods and unsupervised model adaptation methods on DomainNet, Office-Home and Office-31 datasets.
  arXiv  Detail & Related papers  (2021-07-07T04:14:24Z)
• The Power of Sampling: Dimension-free Risk Bounds in Private ERM [25.676350220943274]
  We show that our algorithm can achieve rank-dependent risk bounds for non-smooth convex objectives using only zeroth order oracles. This highlights the power of sampling in differential privacy.
  arXiv  Detail & Related papers  (2021-05-28T07:28:24Z)
• Heterogeneous Risk Minimization [25.5458915855661]
  Invariant learning methods for out-of-distribution generalization have been proposed by leveraging multiple training environments to find invariant relationships. Modern datasets are assembled by merging data from multiple sources without explicit source labels. We propose Heterogeneous Risk Minimization (HRM) framework to achieve joint learning of latent heterogeneity among the data and invariant relationship.
  arXiv  Detail & Related papers  (2021-05-09T02:51:36Z)
• Meta-Learned Invariant Risk Minimization [12.6484257912092]
  Empirical Risk Minimization (ERM) based machine learning algorithms have suffered from weak generalization performance on data obtained from out-of-distribution (OOD) In this paper, we propose a novel meta-learning based approach for IRM. We show that our algorithm not only has better OOD generalization performance than IRMv1 and all IRM variants, but also addresses the weakness of IRMv1 with improved stability.
  arXiv  Detail & Related papers  (2021-03-24T02:52:48Z)
• Treatment Effect Estimation using Invariant Risk Minimization [32.9769365726994]
  In this work, we propose a new way to estimate the causal individual treatment effect (ITE) using the domain generalization framework of invariant risk minimization (IRM) We propose an IRM-based ITE estimator aimed at tackling treatment assignment bias when there is little support overlap between the control group and the treatment group. We show gains over classical regression approaches to ITE estimation in settings when support mismatch is more pronounced.
  arXiv  Detail & Related papers  (2021-03-13T20:42:04Z)
• Empirical or Invariant Risk Minimization? A Sample Complexity Perspective [49.43806345820883]
  It is unclear when in-variant risk generalization (IRM) should be preferred over the widely-employed empirical risk minimization (ERM) framework. We find that depending on the type of data generation mechanism, the two approaches might have very different finite sample and behavior. We further investigate how different factors -- the number of environments, complexity of the model, and IRM penalty weight -- impact the sample complexity of IRM in relation to its distance from the OOD solutions.
  arXiv  Detail & Related papers  (2020-10-30T17:55:30Z)
• The Risks of Invariant Risk Minimization [52.7137956951533]
  Invariant Risk Minimization is an objective based on the idea for learning deep, invariant features of data. We present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model. We show that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
  arXiv  Detail & Related papers  (2020-10-12T14:54:32Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• Unsupervised Domain Adaptation in Person re-ID via k-Reciprocal Clustering and Large-Scale Heterogeneous Environment Synthesis [76.46004354572956]
  We introduce an unsupervised domain adaptation approach for person re-identification. Experimental results show that the proposed ktCUDA and SHRED approach achieves an average improvement of +5.7 mAP in re-identification performance.
  arXiv  Detail & Related papers  (2020-01-14T17:43:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.