Minimax Optimal Transfer Learning for Kernel-based Nonparametric Regression

• URL: http://arxiv.org/abs/2310.13966v1
• Date: Sat, 21 Oct 2023 10:55:31 GMT
• Title: Minimax Optimal Transfer Learning for Kernel-based Nonparametric Regression
• Authors: Chao Wang, Caixing Wang, Xin He, and Xingdong Feng
• Abstract summary: This paper focuses on investigating the transfer learning problem within the context of nonparametric regression. The aim is to bridge the gap between practical effectiveness and theoretical guarantees.
• Score: 18.240776405802205
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, transfer learning has garnered significant attention in the machine learning community. Its ability to leverage knowledge from related studies to improve generalization performance in a target study has made it highly appealing. This paper focuses on investigating the transfer learning problem within the context of nonparametric regression over a reproducing kernel Hilbert space. The aim is to bridge the gap between practical effectiveness and theoretical guarantees. We specifically consider two scenarios: one where the transferable sources are known and another where they are unknown. For the known transferable source case, we propose a two-step kernel-based estimator by solely using kernel ridge regression. For the unknown case, we develop a novel method based on an efficient aggregation algorithm, which can automatically detect and alleviate the effects of negative sources. This paper provides the statistical properties of the desired estimators and establishes the minimax optimal rate. Through extensive numerical experiments on synthetic data and real examples, we validate our theoretical findings and demonstrate the effectiveness of our proposed method.

Related papers

• Observationally Informed Adaptive Causal Experimental Design [55.998153710215654]
  We propose Active Residual Learning, a new paradigm that leverages the observational model as a foundational prior.<n>This approach shifts the experimental focus from learning target causal quantities from scratch to efficiently estimating the residuals required to correct observational bias.<n> Experiments on synthetic and semi-synthetic benchmarks demonstrate that R-Design significantly outperforms baselines.
  arXiv  Detail & Related papers  (2026-03-04T06:52:37Z)
• Why Do Unlearnable Examples Work: A Novel Perspective of Mutual Information [55.75102049412629]
  We show that effective unlearnable examples always decrease mutual information between clean features and poisoned features.<n>We propose a novel unlearnable method called Mutual Information Unlearnable Examples (MI-UE)<n>Our approach significantly outperforms the previous methods, even under defense mechanisms.
  arXiv  Detail & Related papers  (2026-03-04T04:53:29Z)
• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Wasserstein Transfer Learning [6.602088845993411]
  We introduce a novel transfer learning framework for regression models whose outputs are probability distributions residing in the Wasserstein space.<n>We propose an estimator with provable convergence rates, quantifying the impact of domain similarity on transfer efficiency.<n>For cases where the informative subset is unknown, we develop a data-driven transfer learning procedure designed to mitigate negative transfer.
  arXiv  Detail & Related papers  (2025-05-23T02:38:03Z)
• Transfer Learning of CATE with Kernel Ridge Regression [4.588222946914528]
  We propose a novel method for overlap-adaptive transfer learning of conditional average treatment effect (CATE) using kernel ridge regression (KRR) We provide a theoretical justification for our method through sharp non-asymptotic MSE bounds, highlighting its adaptivity to both weak overlaps and the complexity of CATE function.
  arXiv  Detail & Related papers  (2025-02-17T01:07:45Z)
• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n> PPCI estimates the treatment effect in a target experiment with unlabeled factual outcomes, retrievable zero-shot from a pre-trained model.<n>We validate our method on synthetic and real-world scientific data, offering solutions to instances not solvable by vanilla Empirical Risk Minimization.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Model-Robust and Adaptive-Optimal Transfer Learning for Tackling Concept Shifts in Nonparametric Regression [7.243632426715939]
  We present a transfer learning procedure that is robust against model misspecification while adaptively attaining optimality. We derive the adaptive convergence rates of the excess risk for specifying Gaussian kernels in a prevalent class of hypothesis transfer learning algorithms.
  arXiv  Detail & Related papers  (2025-01-18T20:33:37Z)
• A Kernel Perspective on Distillation-based Collaborative Learning [8.971234046933349]
  We propose a nonparametric collaborative learning algorithm that does not directly share local data or models in statistically heterogeneous environments. Inspired by our theoretical results, we also propose a practical distillation-based collaborative learning algorithm based on neural network architecture.
  arXiv  Detail & Related papers  (2024-10-23T06:40:13Z)
• Center-Sensitive Kernel Optimization for Efficient On-Device Incremental Learning [88.78080749909665]
  Current on-device training methods just focus on efficient training without considering the catastrophic forgetting. This paper proposes a simple but effective edge-friendly incremental learning framework. Our method achieves average accuracy boost of 38.08% with even less memory and approximate computation.
  arXiv  Detail & Related papers  (2024-06-13T05:49:29Z)
• Knowledge Transfer across Multiple Principal Component Analysis Studies [8.602833477729899]
  We propose a two-step transfer learning algorithm to extract useful information from multiple source principal component analysis (PCA) studies. In the first step, we integrate the shared subspace information across multiple studies by a proposed method named as Grassmannian barycenter. The resulting estimator for the shared subspace from the first step is further utilized to estimate the target private subspace.
  arXiv  Detail & Related papers  (2024-03-12T09:15:12Z)
• Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling [53.7117640028211]
  We present a geometry-constrained probabilistic modeling treatment to resolve the identified issues. We incorporate a suite of critical geometric properties to impose proper constraints on the layout of constructed embedding space. A spectral graph-theoretic method is devised to estimate the number of potential novel classes.
  arXiv  Detail & Related papers  (2024-03-02T00:56:05Z)
• Transfer Learning for Nonparametric Regression: Non-asymptotic Minimax Analysis and Adaptive Procedure [5.303044915173525]
  We develop a novel estimator called the confidence thresholding estimator, which is shown to achieve the minimax optimal risk up to a logarithmic factor. We then propose a data-driven algorithm that adaptively achieves the minimax risk up to a logarithmic factor across a wide range of parameter spaces.
  arXiv  Detail & Related papers  (2024-01-22T16:24:04Z)
• Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels [57.46832672991433]
  We propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS) We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises. We develop an expectation-propagation expectation-maximization algorithm for efficient posterior inference and function estimation.
  arXiv  Detail & Related papers  (2023-10-09T03:55:09Z)
• Estimation and inference for transfer learning with high-dimensional quantile regression [3.4510296013600374]
  We propose a transfer learning procedure in the framework of high-dimensional quantile regression models. We establish error bounds of transfer learning estimator based on delicately selected transferable source domains. By adopting data-splitting technique, we advocate a transferability detection approach that guarantees to circumvent negative transfer.
  arXiv  Detail & Related papers  (2022-11-26T14:40:19Z)
• On the Benefits of Large Learning Rates for Kernel Methods [110.03020563291788]
  We show that a phenomenon can be precisely characterized in the context of kernel methods. We consider the minimization of a quadratic objective in a separable Hilbert space, and show that with early stopping, the choice of learning rate influences the spectral decomposition of the obtained solution.
  arXiv  Detail & Related papers  (2022-02-28T13:01:04Z)
• Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations [76.82124752950148]
  We develop a convenient gradient-based method for selecting the data augmentation. We use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective.
  arXiv  Detail & Related papers  (2022-02-22T02:51:11Z)
• Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning [57.88785630755165]
  Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data. We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class. For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
  arXiv  Detail & Related papers  (2021-06-03T09:50:13Z)
• Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms [91.3755431537592]
  We analyze four broad meta-learning strategies which rely on plug-in estimation and pseudo-outcome regression. We highlight how this theoretical reasoning can be used to guide principled algorithm design and translate our analyses into practice.
  arXiv  Detail & Related papers  (2021-01-26T17:11:40Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.