PGrad: Learning Principal Gradients For Domain Generalization

• URL: http://arxiv.org/abs/2305.01134v1
• Date: Tue, 2 May 2023 00:48:24 GMT
• Title: PGrad: Learning Principal Gradients For Domain Generalization
• Authors: Zhe Wang, Jake Grigsby, Yanjun Qi
• Abstract summary: We develop a novel DG training strategy, we call PGrad, to learn a robust gradient direction, improving models' generalization ability on unseen domains. PGrad's gradient design forces the DG training to ignore domain-dependent noise signals and updates all training domains with a robust direction. PGrad achieves competitive results across seven datasets, demonstrating its efficacy across both synthetic and real-world distributional shifts.
• Score: 14.134043376245165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning models fail to perform when facing out-of-distribution (OOD) domains, a challenging task known as domain generalization (DG). In this work, we develop a novel DG training strategy, we call PGrad, to learn a robust gradient direction, improving models' generalization ability on unseen domains. The proposed gradient aggregates the principal directions of a sampled roll-out optimization trajectory that measures the training dynamics across all training domains. PGrad's gradient design forces the DG training to ignore domain-dependent noise signals and updates all training domains with a robust direction covering main components of parameter dynamics. We further improve PGrad via bijection-based computational refinement and directional plus length-based calibrations. Our theoretical proof connects PGrad to the spectral analysis of Hessian in training neural networks. Experiments on DomainBed and WILDS benchmarks demonstrate that our approach effectively enables robust DG optimization and leads to smoothly decreased loss curves. Empirically, PGrad achieves competitive results across seven datasets, demonstrating its efficacy across both synthetic and real-world distributional shifts. Code is available at https://github.com/QData/PGrad.

Related papers

• Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning [26.915055027485465]
  We study offline off-dynamics reinforcement learning (RL) to enhance policy learning in a target domain with limited data. Our approach centers on return-conditioned supervised learning (RCSL), particularly focusing on the decision transformer (DT) We propose the Return Augmented Decision Transformer (RADT) method, where we augment the return in the source domain by aligning its distribution with that in the target domain.
  arXiv  Detail & Related papers  (2024-10-30T20:46:26Z)
• Shape Guided Gradient Voting for Domain Generalization [20.593708375868893]
  We propose a Shape Guided Gradient Voting (SGGV) method for domain generalization. Firstly, we introduce shape prior via extra inputs of the network to guide gradient descending towards a shape-biased direction. Secondly, we propose a new gradient voting strategy to remove the outliers for robust optimization.
  arXiv  Detail & Related papers  (2023-06-19T09:54:37Z)
• Stochastic Re-weighted Gradient Descent via Distributionally Robust Optimization [14.23697277904244]
  We present Reweighted Gradient Descent (RGD), a novel optimization technique that improves the performance of deep neural networks through dynamic sample re-weighting. We demonstrate the effectiveness of RGD on various learning tasks, including supervised learning, meta-learning, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2023-06-15T15:58:04Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Learning to Augment via Implicit Differentiation for Domain Generalization [107.9666735637355]
  Domain generalization (DG) aims to overcome the problem by leveraging multiple source domains to learn a domain-generalizable model. In this paper, we propose a novel augmentation-based DG approach, dubbed AugLearn. AugLearn shows effectiveness on three standard DG benchmarks, PACS, Office-Home and Digits-DG.
  arXiv  Detail & Related papers  (2022-10-25T18:51:51Z)
• On Certifying and Improving Generalization to Unseen Domains [87.00662852876177]
  Domain Generalization aims to learn models whose performance remains high on unseen domains encountered at test-time. It is challenging to evaluate DG algorithms comprehensively using a few benchmark datasets. We propose a universal certification framework that can efficiently certify the worst-case performance of any DG method.
  arXiv  Detail & Related papers  (2022-06-24T16:29:43Z)
• Adaptive Trajectory Prediction via Transferable GNN [74.09424229172781]
  We propose a novel Transferable Graph Neural Network (T-GNN) framework, which jointly conducts trajectory prediction as well as domain alignment in a unified framework. Specifically, a domain invariant GNN is proposed to explore the structural motion knowledge where the domain specific knowledge is reduced. An attention-based adaptive knowledge learning module is further proposed to explore fine-grained individual-level feature representation for knowledge transfer.
  arXiv  Detail & Related papers  (2022-03-09T21:08:47Z)
• Towards Principled Disentanglement for Domain Generalization [90.9891372499545]
  A fundamental challenge for machine learning models is generalizing to out-of-distribution (OOD) data. We first formalize the OOD generalization problem as constrained optimization, called Disentanglement-constrained Domain Generalization (DDG) Based on the transformation, we propose a primal-dual algorithm for joint representation disentanglement and domain generalization.
  arXiv  Detail & Related papers  (2021-11-27T07:36:32Z)
• Amortized Prompt: Lightweight Fine-Tuning for CLIP in Domain Generalization [25.367775241988618]
  Domain generalization is a difficult transfer learning problem aiming to learn a generalizable model to unseen domains. Recent massive pre-trained models such as CLIP and GPT-3 have been shown to be robust to many distribution shifts. We propose AP (Amortized Prompt) as a novel approach for domain inference in the form of prompt generation.
  arXiv  Detail & Related papers  (2021-11-25T00:25:54Z)
• Reappraising Domain Generalization in Neural Networks [8.06370138649329]
  Domain generalization (DG) of machine learning algorithms is defined as their ability to learn a domain agnostic hypothesis from multiple training distributions. We find that a straightforward Empirical Risk Minimization (ERM) baseline consistently outperforms existing DG methods. We propose a classwise-DG formulation, where for each class, we randomly select one of the domains and keep it aside for testing.
  arXiv  Detail & Related papers  (2021-10-15T10:06:40Z)
• Supervised Domain Adaptation using Graph Embedding [86.3361797111839]
  Domain adaptation methods assume that distributions between the two domains are shifted and attempt to realign them. We propose a generic framework based on graph embedding. We show that the proposed approach leads to a powerful Domain Adaptation framework.
  arXiv  Detail & Related papers  (2020-03-09T12:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.