Transfer-Learning-Based Autotuning Using Gaussian Copula

• URL: http://arxiv.org/abs/2401.04669v1
• Date: Tue, 9 Jan 2024 16:52:57 GMT
• Title: Transfer-Learning-Based Autotuning Using Gaussian Copula
• Authors: Thomas Randall (1), Jaehoon Koo (2), Brice Videau (3), Michael Kruse (3), Xingfu Wu (3), Paul Hovland (3), Mary Hall (4), Rong Ge (1), Prasanna Balaprakash (5) ((1) Clemson University, (2) Hanyang University, (3) Argonne National Laboratory, (4) University of Utah, (5) Oak Ridge National Laboratory)
• Abstract summary: We introduce the first generative TL-based autotuning approach based on the Gaussian copula (GC) We find that the GC is capable of achieving 64.37% of peak few-shot performance in its first evaluation. Furthermore, the GC model can determine a few-shot transfer budget that yields up to 33.39$times$ speedup.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As diverse high-performance computing (HPC) systems are built, many opportunities arise for applications to solve larger problems than ever before. Given the significantly increased complexity of these HPC systems and application tuning, empirical performance tuning, such as autotuning, has emerged as a promising approach in recent years. Despite its effectiveness, autotuning is often a computationally expensive approach. Transfer learning (TL)-based autotuning seeks to address this issue by leveraging the data from prior tuning. Current TL methods for autotuning spend significant time modeling the relationship between parameter configurations and performance, which is ineffective for few-shot (that is, few empirical evaluations) tuning on new tasks. We introduce the first generative TL-based autotuning approach based on the Gaussian copula (GC) to model the high-performing regions of the search space from prior data and then generate high-performing configurations for new tasks. This allows a sampling-based approach that maximizes few-shot performance and provides the first probabilistic estimation of the few-shot budget for effective TL-based autotuning. We compare our generative TL approach with state-of-the-art autotuning techniques on several benchmarks. We find that the GC is capable of achieving 64.37% of peak few-shot performance in its first evaluation. Furthermore, the GC model can determine a few-shot transfer budget that yields up to 33.39$\times$ speedup, a dramatic improvement over the 20.58$\times$ speedup using prior techniques.

Related papers

• Controllable Prompt Tuning For Balancing Group Distributional Robustness [53.336515056479705]
  We introduce an optimization scheme to achieve good performance across groups and find a good solution for all without severely sacrificing performance on any of them. We propose Controllable Prompt Tuning (CPT), which couples our approach with prompt-tuning techniques. On spurious correlation benchmarks, our procedures achieve state-of-the-art results across both transformer and non-transformer architectures, as well as unimodal and multimodal data.
  arXiv  Detail & Related papers  (2024-03-05T06:23:55Z)
• AutoFT: Learning an Objective for Robust Fine-Tuning [60.641186718253735]
  Foundation models encode rich representations that can be adapted to downstream tasks by fine-tuning. Current approaches to robust fine-tuning use hand-crafted regularization techniques. We propose AutoFT, a data-driven approach for robust fine-tuning.
  arXiv  Detail & Related papers  (2024-01-18T18:58:49Z)
• Surrogate-based Autotuning for Randomized Sketching Algorithms in Regression Problems [44.54726768134471]
  We show how a surrogate-based autotuning approach can be used to address fundamental problems of parameter selection in RandNLA algorithms. Our results show that our surrogate-based autotuning approach can achieve near-optimal performance with much less tuning cost than a random search.
  arXiv  Detail & Related papers  (2023-08-30T02:50:54Z)
• E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuning [55.50908600818483]
  Fine-tuning large-scale pretrained vision models for new tasks has become increasingly parameter-intensive. We propose an Effective and Efficient Visual Prompt Tuning (E2VPT) approach for large-scale transformer-based model adaptation. Our approach outperforms several state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2023-07-25T19:03:21Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training [0.0]
  We propose a practical method for robustly tuning large models. CarBS performs local search around the performance-cost frontier. Among our results, we effectively solve the entire ProcGen benchmark just by tuning a simple baseline.
  arXiv  Detail & Related papers  (2023-06-13T18:22:24Z)
• Hyper-Parameter Auto-Tuning for Sparse Bayesian Learning [72.83293818245978]
  We design and learn a neural network (NN)-based auto-tuner for hyper- parameter tuning in sparse Bayesian learning. We show that considerable improvement in convergence rate and recovery performance can be achieved.
  arXiv  Detail & Related papers  (2022-11-09T12:34:59Z)
• Amortized Auto-Tuning: Cost-Efficient Transfer Optimization for Hyperparameter Recommendation [83.85021205445662]
  We propose an instantiation--amortized auto-tuning (AT2) to speed up tuning of machine learning models. We conduct a thorough analysis of the multi-task multi-fidelity Bayesian optimization framework, which leads to the best instantiation--amortized auto-tuning (AT2)
  arXiv  Detail & Related papers  (2021-06-17T00:01:18Z)
• FIST: A Feature-Importance Sampling and Tree-Based Method for Automatic Design Flow Parameter Tuning [27.08970520268831]
  We introduce a machine learning-based automatic parameter tuning methodology that aims to find the best design quality with a limited number of trials. We leverage a state-of-the-art XGBoost model and propose a novel dynamic tree technique to overcome overfitting. Experimental results on benchmark circuits show that our approach achieves 25% improvement in design quality or reduction in sampling cost.
  arXiv  Detail & Related papers  (2020-11-26T23:13:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.