AutoTransfer: AutoML with Knowledge Transfer -- An Application to Graph Neural Networks

• URL: http://arxiv.org/abs/2303.07669v1
• Date: Tue, 14 Mar 2023 07:23:16 GMT
• Title: AutoTransfer: AutoML with Knowledge Transfer -- An Application to Graph Neural Networks
• Authors: Kaidi Cao, Jiaxuan You, Jiaju Liu, Jure Leskovec
• Abstract summary: AutoML techniques consider each task independently from scratch, leading to high computational cost. Here we propose AutoTransfer, an AutoML solution that improves search efficiency by transferring the prior architectural design knowledge to the novel task of interest.
• Score: 75.11008617118908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AutoML has demonstrated remarkable success in finding an effective neural architecture for a given machine learning task defined by a specific dataset and an evaluation metric. However, most present AutoML techniques consider each task independently from scratch, which requires exploring many architectures, leading to high computational cost. Here we propose AutoTransfer, an AutoML solution that improves search efficiency by transferring the prior architectural design knowledge to the novel task of interest. Our key innovation includes a task-model bank that captures the model performance over a diverse set of GNN architectures and tasks, and a computationally efficient task embedding that can accurately measure the similarity among different tasks. Based on the task-model bank and the task embeddings, we estimate the design priors of desirable models of the novel task, by aggregating a similarity-weighted sum of the top-K design distributions on tasks that are similar to the task of interest. The computed design priors can be used with any AutoML search algorithm. We evaluate AutoTransfer on six datasets in the graph machine learning domain. Experiments demonstrate that (i) our proposed task embedding can be computed efficiently, and that tasks with similar embeddings have similar best-performing architectures; (ii) AutoTransfer significantly improves search efficiency with the transferred design priors, reducing the number of explored architectures by an order of magnitude. Finally, we release GNN-Bank-101, a large-scale dataset of detailed GNN training information of 120,000 task-model combinations to facilitate and inspire future research.

Related papers

• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• Transferability Metrics for Object Detection [0.0]
  Transfer learning aims to make the most of existing pre-trained models to achieve better performance on a new task in limited data scenarios. We extend transferability metrics to object detection using ROI-Align and TLogME. We show that TLogME provides a robust correlation with transfer performance and outperforms other transferability metrics on local and global level features.
  arXiv  Detail & Related papers  (2023-06-27T08:49:31Z)
• Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search [96.31315520244605]
  Arch-Graph is a transferable NAS method that predicts task-specific optimal architectures. We show Arch-Graph's transferability and high sample efficiency across numerous tasks. It is able to find top 0.16% and 0.29% architectures on average on two search spaces under the budget of only 50 models.
  arXiv  Detail & Related papers  (2022-04-12T16:46:06Z)
• A Tree-Structured Multi-Task Model Recommender [25.445073413243925]
  Tree-structured multi-task architectures have been employed to tackle multiple vision tasks in the context of multi-task learning (MTL) This paper proposes a recommender that automatically suggests tree-structured multi-task architectures that could achieve a high task performance while meeting a user-specified computation budget without performing model training. Extensive evaluations on popular MTL benchmarks show that the recommended architectures could achieve competitive task accuracy and computation efficiency compared with state-of-the-art MTL methods.
  arXiv  Detail & Related papers  (2022-03-10T00:09:43Z)
• DAAS: Differentiable Architecture and Augmentation Policy Search [107.53318939844422]
  This work considers the possible coupling between neural architectures and data augmentation and proposes an effective algorithm jointly searching for them. Our approach achieves 97.91% accuracy on CIFAR-10 and 76.6% Top-1 accuracy on ImageNet dataset, showing the outstanding performance of our search algorithm.
  arXiv  Detail & Related papers  (2021-09-30T17:15:17Z)
• Neural Architecture Search From Fr\'echet Task Distance [50.9995960884133]
  We show how the distance between a target task and each task in a given set of baseline tasks can be used to reduce the neural architecture search space for the target task. The complexity reduction in search space for task-specific architectures is achieved by building on the optimized architectures for similar tasks instead of doing a full search without using this side information.
  arXiv  Detail & Related papers  (2021-03-23T20:43:31Z)
• NASirt: AutoML based learning with instance-level complexity information [0.0]
  We present NASirt, an AutoML methodology that finds high accuracy CNN architectures for spectral datasets. Our method performs, in most cases, better than the benchmarks, achieving average accuracy as high as 97.40%.
  arXiv  Detail & Related papers  (2020-08-26T22:21:44Z)
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning [71.90902837008278]
  We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL) In order to adapt to different task combinations, we disentangle the GP-MTL networks into single-task backbones. We also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures.
  arXiv  Detail & Related papers  (2020-03-31T09:49:14Z)
• NeurAll: Towards a Unified Visual Perception Model for Automated Driving [8.49826472556323]
  We propose a joint multi-task network design for learning several tasks simultaneously. The main bottleneck in automated driving systems is the limited processing power available on deployment hardware.
  arXiv  Detail & Related papers  (2019-02-10T12:45:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.