Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks

• URL: http://arxiv.org/abs/2501.14012v2
• Date: Sat, 08 Feb 2025 12:02:02 GMT
• Title: Transfer Learning of Surrogate Models via Domain Affine Transformation Across Synthetic and Real-World Benchmarks
• Authors: Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang,
• Abstract summary: Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. This study focuses on transferring non-differentiable surrogate models from a source function to a target function. We assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target.
• Score: 4.515998639772672
• License:
• Abstract: Surrogate models are frequently employed as efficient substitutes for the costly execution of real-world processes. However, constructing a high-quality surrogate model often demands extensive data acquisition. A solution to this issue is to transfer pre-trained surrogate models for new tasks, provided that certain invariances exist between tasks. This study focuses on transferring non-differentiable surrogate models (e.g., random forest) from a source function to a target function, where we assume their domains are related by an unknown affine transformation, using only a limited amount of transfer data points evaluated on the target. Previous research attempts to tackle this challenge for differentiable models, e.g., Gaussian process regression, which minimizes the empirical loss on the transfer data by tuning the affine transformations. In this paper, we extend the previous work to the random forest model and assess its effectiveness on a widely-used artificial problem set - Black-Box Optimization Benchmark (BBOB) testbed, and on four real-world transfer learning problems. The results highlight the significant practical advantages of the proposed method, particularly in reducing both the data requirements and computational costs of training surrogate models for complex real-world scenarios.

Related papers

• Transfer Learning of Surrogate Models: Integrating Domain Warping and Affine Transformations [4.515998639772672]
  Surrogate models provide efficient alternatives to computationally demanding real-world processes. Previous studies have investigated the transfer of differentiable and non-differentiable surrogate models. This paper extends previous research by addressing a broader range of transformations.
  arXiv  Detail & Related papers  (2025-01-30T13:46:48Z)
• Enhancing Polynomial Chaos Expansion Based Surrogate Modeling using a Novel Probabilistic Transfer Learning Strategy [2.980666177064344]
  In black-box simulations, non-intrusive PCE allows the construction of surrogates using a set of simulation response evaluations. We propose to leverage transfer learning whereby knowledge gained through similar PCE surrogate construction tasks is transferred to a new surrogate-construction task.
  arXiv  Detail & Related papers  (2023-12-07T19:16:42Z)
• Informative Data Mining for One-Shot Cross-Domain Semantic Segmentation [84.82153655786183]
  We propose a novel framework called Informative Data Mining (IDM) to enable efficient one-shot domain adaptation for semantic segmentation. IDM provides an uncertainty-based selection criterion to identify the most informative samples, which facilitates quick adaptation and reduces redundant training. Our approach outperforms existing methods and achieves a new state-of-the-art one-shot performance of 56.7%/55.4% on the GTA5/SYNTHIA to Cityscapes adaptation tasks.
  arXiv  Detail & Related papers  (2023-09-25T15:56:01Z)
• Revisiting the Robustness of the Minimum Error Entropy Criterion: A Transfer Learning Case Study [16.07380451502911]
  This paper revisits the robustness of the minimum error entropy criterion to deal with non-Gaussian noises. We investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common.
  arXiv  Detail & Related papers  (2023-07-17T15:38:11Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation [62.889835139583965]
  We introduce an unsupervised auxiliary task of learning an implicit underlying surface representation simultaneously on source and target data. As both domains share the same latent representation, the model is forced to accommodate discrepancies between the two sources of data. Our experiments demonstrate that our method achieves a better performance than the current state of the art, both in real-to-real and synthetic-to-real scenarios.
  arXiv  Detail & Related papers  (2023-04-06T17:36:23Z)
• Towards Estimating Transferability using Hard Subsets [25.86053764521497]
  We propose HASTE, a new strategy to estimate the transferability of a source model to a particular target task using only a harder subset of target data. We show that HASTE can be used with any existing transferability metric to improve their reliability. Our experimental results across multiple source model architectures, target datasets, and transfer learning tasks show that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics.
  arXiv  Detail & Related papers  (2023-01-17T14:50:18Z)
• Transfer learning with affine model transformation [18.13383101189326]
  This paper presents a general class of transfer learning regression called affine model transfer. It is shown that the affine model transfer broadly encompasses various existing methods, including the most common procedure based on neural feature extractors.
  arXiv  Detail & Related papers  (2022-10-18T10:50:24Z)
• Frustratingly Easy Transferability Estimation [64.42879325144439]
  We propose a simple, efficient, and effective transferability measure named TransRate. TransRate measures the transferability as the mutual information between the features of target examples extracted by a pre-trained model and labels of them. Despite its extraordinary simplicity in 10 lines of codes, TransRate performs remarkably well in extensive evaluations on 22 pre-trained models and 16 downstream tasks.
  arXiv  Detail & Related papers  (2021-06-17T10:27:52Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.