Transfer Learning of Surrogate Models: Integrating Domain Warping and Affine Transformations

• URL: http://arxiv.org/abs/2501.18344v1
• Date: Thu, 30 Jan 2025 13:46:48 GMT
• Title: Transfer Learning of Surrogate Models: Integrating Domain Warping and Affine Transformations
• Authors: Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang,
• Abstract summary: Surrogate models provide efficient alternatives to computationally demanding real-world processes. Previous studies have investigated the transfer of differentiable and non-differentiable surrogate models. This paper extends previous research by addressing a broader range of transformations.
• Score: 4.515998639772672
• License:
• Abstract: Surrogate models provide efficient alternatives to computationally demanding real-world processes but often require large datasets for effective training. A promising solution to this limitation is the transfer of pre-trained surrogate models to new tasks. Previous studies have investigated the transfer of differentiable and non-differentiable surrogate models, typically assuming an affine transformation between the source and target functions. This paper extends previous research by addressing a broader range of transformations, including linear and nonlinear variations. Specifically, we consider the combination of an unknown input warping, such as one modelled by the beta cumulative distribution function, with an unspecified affine transformation. Our approach achieves transfer learning by employing a limited number of data points from the target task to optimize these transformations, minimizing empirical loss on the transfer dataset. We validate the proposed method on the widely used Black-Box Optimization Benchmark (BBOB) testbed and a real-world transfer learning task from the automobile industry. The results underscore the significant advantages of the approach, revealing that the transferred surrogate significantly outperforms both the original surrogate and the one built from scratch using the transfer dataset, particularly in data-scarce scenarios.

Related papers

• Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks [4.515998639772672]
  Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. This study focuses on transferring non-differentiable surrogate models from a source function to a target function. We assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target.
  arXiv  Detail & Related papers  (2025-01-23T18:44:25Z)
• Bayesian Inverse Transfer in Evolutionary Multiobjective Optimization [29.580786235313987]
  We introduce the first Inverse Transfer Multiobjective (invTrEMO) InvTrEMO harnesses the common objective functions in many prevalent areas, even when decision spaces do not precisely align between tasks. InvTrEMO yields high-precision inverse models as a significant byproduct, enabling the generation of tailored solutions on-demand.
  arXiv  Detail & Related papers  (2023-12-22T14:12:18Z)
• Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasets [9.510327380529892]
  We propose a simple approach to pre-train a surrogate, which is a Gaussian process (GP) with a kernel defined on deep features learned from a Transformer-based encoder. Experiments on both synthetic and real benchmark problems demonstrate the effectiveness of our proposed pre-training and transfer BO strategy.
  arXiv  Detail & Related papers  (2023-08-09T01:56:10Z)
• Towards Estimating Transferability using Hard Subsets [25.86053764521497]
  We propose HASTE, a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. We show that HASTE can be used with any existing transferability metric to improve their reliability. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics.
  arXiv  Detail & Related papers  (2023-01-17T14:50:18Z)
• Estimation and inference for transfer learning with high-dimensional quantile regression [3.4510296013600374]
  We propose a transfer learning procedure in the framework of high-dimensional quantile regression models. We establish error bounds of transfer learning estimator based on delicately selected transferable source domains. By adopting data-splitting technique, we advocate a transferability detection approach that guarantees to circumvent negative transfer.
  arXiv  Detail & Related papers  (2022-11-26T14:40:19Z)
• Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations [76.82124752950148]
  We develop a convenient gradient-based method for selecting the data augmentation. We use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective.
  arXiv  Detail & Related papers  (2022-02-22T02:51:11Z)
• How Well Do Sparse Imagenet Models Transfer? [75.98123173154605]
  Transfer learning is a classic paradigm by which models pretrained on large "upstream" datasets are adapted to yield good results on "downstream" datasets. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset. We show that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities.
  arXiv  Detail & Related papers  (2021-11-26T11:58:51Z)
• Frustratingly Easy Transferability Estimation [64.42879325144439]
  We propose a simple, efficient, and effective transferability measure named TransRate. TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.
  arXiv  Detail & Related papers  (2021-06-17T10:27:52Z)
• Transformer-Based Source-Free Domain Adaptation [134.67078085569017]
  We study the task of source-free domain adaptation (SFDA), where the source data are not available during target adaptation. We propose a generic and effective framework based on Transformer, named TransDA, for learning a generalized model for SFDA.
  arXiv  Detail & Related papers  (2021-05-28T23:06:26Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.