Knowledge-Guided Wasserstein Distributionally Robust Optimization

• URL: http://arxiv.org/abs/2502.08146v1
• Date: Wed, 12 Feb 2025 06:09:27 GMT
• Title: Knowledge-Guided Wasserstein Distributionally Robust Optimization
• Authors: Zitao Wang, Ziyuan Wang, Molei Liu, Nian Si,
• Abstract summary: Transfer learning is a popular strategy to leverage external knowledge and improve statistical efficiency. We propose a knowledge-guided Wasserstein Distributionally Robust Optimization framework to overcome the conservativeness of vanilla WDRO. Our method constructs smaller Wasserstein ambiguity sets by controlling the transportation along directions informed by the source knowledge.
• Score: 9.980826279322734
• License:
• Abstract: Transfer learning is a popular strategy to leverage external knowledge and improve statistical efficiency, particularly with a limited target sample. We propose a novel knowledge-guided Wasserstein Distributionally Robust Optimization (KG-WDRO) framework that adaptively incorporates multiple sources of external knowledge to overcome the conservativeness of vanilla WDRO, which often results in overly pessimistic shrinkage toward zero. Our method constructs smaller Wasserstein ambiguity sets by controlling the transportation along directions informed by the source knowledge. This strategy can alleviate perturbations on the predictive projection of the covariates and protect against information loss. Theoretically, we establish the equivalence between our WDRO formulation and the knowledge-guided shrinkage estimation based on collinear similarity, ensuring tractability and geometrizing the feasible set. This also reveals a novel and general interpretation for recent shrinkage-based transfer learning approaches from the perspective of distributional robustness. In addition, our framework can adjust for scaling differences in the regression models between the source and target and accommodates general types of regularization such as lasso and ridge. Extensive simulations demonstrate the superior performance and adaptivity of KG-WDRO in enhancing small-sample transfer learning.

Related papers

• Model-Robust and Adaptive-Optimal Transfer Learning for Tackling Concept Shifts in Nonparametric Regression [7.243632426715939]
  We present a transfer learning procedure that is robust against model misspecification while adaptively attaining optimality. We derive the adaptive convergence rates of the excess risk for specifying Gaussian kernels in a prevalent class of hypothesis transfer learning algorithms.
  arXiv  Detail & Related papers  (2025-01-18T20:33:37Z)
• Transferable Adversarial Attacks on SAM and Its Downstream Models [87.23908485521439]
  This paper explores the feasibility of adversarial attacking various downstream models fine-tuned from the segment anything model (SAM) To enhance the effectiveness of the adversarial attack towards models fine-tuned on unknown datasets, we propose a universal meta-initialization (UMI) algorithm.
  arXiv  Detail & Related papers  (2024-10-26T15:04:04Z)
• Towards Robust Recommendation via Decision Boundary-aware Graph Contrastive Learning [25.514007761856632]
  graph contrastive learning (GCL) has received increasing attention in recommender systems due to its effectiveness in reducing bias caused by data sparsity. We argue that these methods struggle to balance between semantic invariance and view hardness across the dynamic training process. We propose a novel GCL-based recommendation framework RGCL, which effectively maintains the semantic invariance of contrastive pairs and dynamically adapts as the model capability evolves.
  arXiv  Detail & Related papers  (2024-07-14T13:03:35Z)
• TransFusion: Covariate-Shift Robust Transfer Learning for High-Dimensional Regression [11.040033344386366]
  We propose a two-step method with a novel fused-regularizer to improve the learning performance on a target task with limited samples. Nonasymptotic bound is provided for the estimation error of the target model. We extend the method to a distributed setting, allowing for a pretraining-finetuning strategy.
  arXiv  Detail & Related papers  (2024-04-01T14:58:16Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• Robust Transfer Learning with Unreliable Source Data [13.276850367115333]
  We introduce a novel quantity called the ''ambiguity level'' that measures the discrepancy between the target and source regression functions. We propose a simple transfer learning procedure, and establish a general theorem that shows how this new quantity is related to the transferability of learning.
  arXiv  Detail & Related papers  (2023-10-06T21:50:21Z)
• STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning [111.75423966239092]
  We propose an exploration incentive in terms of the integral probability metric (IPM) between a current estimate of the transition model and the unknown optimal. Based on KSD, we develop a novel algorithm algo: textbfSTEin information dirtextbfEcted exploration for model-based textbfReinforcement LearntextbfING.
  arXiv  Detail & Related papers  (2023-01-28T00:49:28Z)
• R\'enyiCL: Contrastive Representation Learning with Skew R\'enyi Divergence [78.15455360335925]
  We present a new robust contrastive learning scheme, coined R'enyiCL, which can effectively manage harder augmentations. Our method is built upon the variational lower bound of R'enyi divergence. We show that R'enyi contrastive learning objectives perform innate hard negative sampling and easy positive sampling simultaneously.
  arXiv  Detail & Related papers  (2022-08-12T13:37:05Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach [150.8920602230832]
  We propose a framework for learning calibrated uncertainties under domain shifts. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks.
  arXiv  Detail & Related papers  (2020-10-08T02:10:54Z)
• Principled learning method for Wasserstein distributionally robust optimization with local perturbations [21.611525306059985]
  Wasserstein distributionally robust optimization (WDRO) attempts to learn a model that minimizes the local worst-case risk in the vicinity of the empirical data distribution. We propose a minimizer based on a novel approximation theorem and provide the corresponding risk consistency results. Our results show that the proposed method achieves significantly higher accuracy than baseline models on noisy datasets.
  arXiv  Detail & Related papers  (2020-06-05T09:32:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.