Learning Representations that Support Robust Transfer of Predictors

• URL: http://arxiv.org/abs/2110.09940v1
• Date: Tue, 19 Oct 2021 13:00:37 GMT
• Title: Learning Representations that Support Robust Transfer of Predictors
• Authors: Yilun Xu, Tommi Jaakkola
• Abstract summary: We introduce a robust estimation criterion -- transfer risk -- that is specifically geared towards optimizing transfer to new environments. Although inspired by IRM, we show that transfer risk serves as a better out-of-distribution generalization criterion.
• Score: 5.65658124285176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring generalization to unseen environments remains a challenge. Domain shift can lead to substantially degraded performance unless shifts are well-exercised within the available training environments. We introduce a simple robust estimation criterion -- transfer risk -- that is specifically geared towards optimizing transfer to new environments. Effectively, the criterion amounts to finding a representation that minimizes the risk of applying any optimal predictor trained on one environment to another. The transfer risk essentially decomposes into two terms, a direct transfer term and a weighted gradient-matching term arising from the optimality of per-environment predictors. Although inspired by IRM, we show that transfer risk serves as a better out-of-distribution generalization criterion, both theoretically and empirically. We further demonstrate the impact of optimizing such transfer risk on two controlled settings, each representing a different pattern of environment shift, as well as on two real-world datasets. Experimentally, the approach outperforms baselines across various out-of-distribution generalization tasks. Code is available at \url{https://github.com/Newbeeer/TRM}.

Related papers

• Transfer Learning Through Conditional Quantile Matching [3.86972243789112]
  We introduce a transfer learning framework for regression that leverages heterogeneous source domains to improve predictive performance in a data-scarce target domain.<n>Our approach learns a conditional generative model separately for each source domain and calibrates the generated responses to the target domain via conditional quantile matching.
  arXiv  Detail & Related papers  (2026-02-02T17:19:55Z)
• Environment-Conditioned Tail Reweighting for Total Variation Invariant Risk Minimization [14.19414103184649]
  Out-of-distribution (OOD) generalization is challenging when models simultaneously encounter correlation shifts across environments.<n>Existing invariant risk minimization (IRM) methods primarily address spurious correlations at the environment level.<n>We propose Environment-Conditioned Tail Reweighting for Total Variation Invariant Risk Minimization (ECTR), a unified framework that augments TV-based invariant learning with environment-conditioned tail reweighting.
  arXiv  Detail & Related papers  (2026-01-30T13:03:04Z)
• RMBRec: Robust Multi-Behavior Recommendation towards Target Behaviors [26.88506691092044]
  We propose Robust Multi-Behavior Recommendation towards Target Behaviors (RMBRec)<n>RMBRec is a robust multi-behavior recommendation framework grounded in an information-theoretic robustness principle.<n>We show that RMBRec outperforms state-of-the-art methods in accuracy and maintains remarkable stability under various noise perturbations.
  arXiv  Detail & Related papers  (2026-01-13T16:34:17Z)
• Optimal Empirical Risk Minimization under Temporal Distribution Shifts [6.374763930914524]
  This paper introduces RIDER, which derives optimally-weighted empirical risk minimization procedures under temporal distribution shifts.<n>We show that RIDER consistently improves out-of-sample predictive performance when applied as a fine-tuning step on the Yearbook dataset.<n>We also show that RIDER outperforms standard weighting strategies in two other real-world tasks: predicting stock market volatility and forecasting ride durations in NYC taxi data.
  arXiv  Detail & Related papers  (2025-07-17T16:53:31Z)
• Robust Invariant Representation Learning by Distribution Extrapolation [3.5051814539447474]
  Invariant risk minimization (IRM) aims to enable out-of-distribution generalization in deep learning.<n>Existing approaches -- including IRMv1 -- adopt penalty-based single-level approximations.<n>A novel framework is proposed that enhances environmental diversity by augmenting the IRM penalty through synthetic distributional shifts.
  arXiv  Detail & Related papers  (2025-05-22T02:03:34Z)
• Towards Context-Aware Domain Generalization: Understanding the Benefits and Limits of Marginal Transfer Learning [1.5320861212113897]
  We formalize the notion of context as a permutation-invariant representation of a set of data points. Empirical analysis shows that our criteria are effective in discerning both favorable and unfavorable scenarios.
  arXiv  Detail & Related papers  (2023-12-15T05:18:07Z)
• Continual Invariant Risk Minimization [46.051656238770086]
  Empirical risk minimization can lead to poor generalization behavior on unseen environments if the learned model does not capture invariant feature representations. Invariant risk minimization (IRM) is a recent proposal for discovering environment-invariant representations.
  arXiv  Detail & Related papers  (2023-10-21T11:44:47Z)
• Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study [16.07380451502911]
  This paper revisits the robustness of the minimum error entropy criterion to deal with non-Gaussian noises. We investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common.
  arXiv  Detail & Related papers  (2023-07-17T15:38:11Z)
• Domain-Specific Risk Minimization for Out-of-Distribution Generalization [104.17683265084757]
  We first establish a generalization bound that explicitly considers the adaptivity gap. We propose effective gap estimation methods for guiding the selection of a better hypothesis for the target. The other method is minimizing the gap directly by adapting model parameters using online target samples.
  arXiv  Detail & Related papers  (2022-08-18T06:42:49Z)
• Grounding Aleatoric Uncertainty in Unsupervised Environment Design [32.00797965770773]
  In partially-observable settings, optimal policies may depend on the ground-truth distribution over aleatoric parameters of the environment. We propose a minimax regret UED method that optimize the ground-truth utility function, even when the underlying training data is biased due to CICS.
  arXiv  Detail & Related papers  (2022-07-11T22:45:29Z)
• On the Equity of Nuclear Norm Maximization in Unsupervised Domain Adaptation [53.29437277730871]
  Nuclear norm has shown the power to enhance the transferability of unsupervised domain adaptation model. Two new losses are proposed to maximize both predictive discriminability and equity, from the class level and the sample level.
  arXiv  Detail & Related papers  (2022-04-12T07:55:47Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Robust Reconfigurable Intelligent Surfaces via Invariant Risk and Causal Representations [55.50218493466906]
  In this paper, the problem of robust reconfigurable intelligent surface (RIS) system design under changes in data distributions is investigated. Using the notion of invariant risk minimization (IRM), an invariant causal representation across multiple environments is used such that the predictor is simultaneously optimal for each environment. A neural network-based solution is adopted to seek the predictor and its performance is validated via simulations against an empirical risk minimization-based design.
  arXiv  Detail & Related papers  (2021-05-04T21:36:31Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
  A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution. We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
  arXiv  Detail & Related papers  (2020-07-08T11:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.