Rapid Model Architecture Adaption for Meta-Learning

• URL: http://arxiv.org/abs/2109.04925v1
• Date: Fri, 10 Sep 2021 15:13:54 GMT
• Title: Rapid Model Architecture Adaption for Meta-Learning
• Authors: Yiren Zhao, Xitong Gao, Ilia Shumailov, Nicolo Fusi, Robert Mullins
• Abstract summary: We show how to rapidly adapt model architectures to new tasks in a few-shot learning setup by integrating Model A Meta Learning (MAML) into the NAS flow. The proposed NAS method (H-Meta-NAS) is hardware-aware and performs computation in the MAML framework. In particular, on the 5-way 1-shot Mini-ImageNet classification task, the proposed method outperforms the best manual baseline by a large margin.
• Score: 5.109810774427172
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Network Architecture Search (NAS) methods have recently gathered much attention. They design networks with better performance and use a much shorter search time compared to traditional manual tuning. Despite their efficiency in model deployments, most NAS algorithms target a single task on a fixed hardware system. However, real-life few-shot learning environments often cover a great number of tasks (T ) and deployments on a wide variety of hardware platforms (H ). The combinatorial search complexity T times H creates a fundamental search efficiency challenge if one naively applies existing NAS methods to these scenarios. To overcome this issue, we show, for the first time, how to rapidly adapt model architectures to new tasks in a many-task many-hardware few-shot learning setup by integrating Model Agnostic Meta Learning (MAML) into the NAS flow. The proposed NAS method (H-Meta-NAS) is hardware-aware and performs optimisation in the MAML framework. H-Meta-NAS shows a Pareto dominance compared to a variety of NAS and manual baselines in popular few-shot learning benchmarks with various hardware platforms and constraints. In particular, on the 5-way 1-shot Mini-ImageNet classification task, the proposed method outperforms the best manual baseline by a large margin (5.21% in accuracy) using 60% less computation.

Related papers

• Multi-objective Differentiable Neural Architecture Search [58.67218773054753]
  We propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics. Our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets.
  arXiv  Detail & Related papers  (2024-02-28T10:09:04Z)
• Meta-prediction Model for Distillation-Aware NAS on Unseen Datasets [55.2118691522524]
  Distillation-aware Neural Architecture Search (DaNAS) aims to search for an optimal student architecture. We propose a distillation-aware meta accuracy prediction model, DaSS (Distillation-aware Student Search), which can predict a given architecture's final performances on a dataset.
  arXiv  Detail & Related papers  (2023-05-26T14:00:35Z)
• Efficient Architecture Search for Diverse Tasks [29.83517145790238]
  We study neural architecture search (NAS) for efficiently solving diverse problems. We introduce DASH, a differentiable NAS algorithm that computes the mixture-of-operations using the Fourier diagonalization of convolution. We evaluate DASH-Bench-360, a suite of ten tasks designed for NAS benchmarking in diverse domains.
  arXiv  Detail & Related papers  (2022-04-15T17:21:27Z)
• Meta-Learning of NAS for Few-shot Learning in Medical Image Applications [10.666687733540668]
  Neural Architecture Search (NAS) has motivated various applications in medical imaging. NAS requires the availability of large annotated data, considerable resources, and pre-defined tasks. We introduce various NAS approaches in medical imaging with different applications such as classification, segmentation, detection, reconstruction.
  arXiv  Detail & Related papers  (2022-03-16T21:21:51Z)
• Across-Task Neural Architecture Search via Meta Learning [1.225795556154044]
  Adequate labeled data and expensive compute resources are the prerequisites for the success of neural architecture search(NAS) It is challenging to apply NAS in meta-learning scenarios with limited compute resources and data. In this paper, an across-task neural architecture search (AT-NAS) is proposed to address the problem through combining gradient-based meta-learning with EA-based NAS.
  arXiv  Detail & Related papers  (2021-10-12T09:07:33Z)
• NAS-Bench-360: Benchmarking Diverse Tasks for Neural Architecture Search [18.9676056830197]
  Most existing neural architecture search (NAS) benchmarks and algorithms prioritize performance on well-studied tasks. We present NAS-Bench-360, a benchmark suite for evaluating state-of-the-art NAS methods for convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2021-10-12T01:13:18Z)
• Elastic Architecture Search for Diverse Tasks with Different Resources [87.23061200971912]
  We study a new challenging problem of efficient deployment for diverse tasks with different resources, where the resource constraint and task of interest corresponding to a group of classes are dynamically specified at testing time. Previous NAS approaches seek to design architectures for all classes simultaneously, which may not be optimal for some individual tasks. We present a novel and general framework, called Elastic Architecture Search (EAS), permitting instant specializations at runtime for diverse tasks with various resource constraints.
  arXiv  Detail & Related papers  (2021-08-03T00:54:27Z)
• Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
  We present a novel approach by connecting model-guided design with NAS (MoD-NAS) MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer. Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-06T14:03:01Z)
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search [6.23453131728063]
  Neural Architecture Search (NAS) methods have been growing in popularity. NAS has been extensively studied in the past few years. Applying NAS to real-world problems still poses significant challenges and is not widely practical. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions.
  arXiv  Detail & Related papers  (2021-01-22T21:13:46Z)
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning [71.90902837008278]
  We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL) In order to adapt to different task combinations, we disentangle the GP-MTL networks into single-task backbones. We also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures.
  arXiv  Detail & Related papers  (2020-03-31T09:49:14Z)
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning [135.27931587381596]
  We propose an efficient and unified NAS framework termed DDPNAS via dynamic distribution pruning. In particular, we first sample architectures from a joint categorical distribution. Then the search space is dynamically pruned and its distribution is updated every few epochs. With the proposed efficient network generation method, we directly obtain the optimal neural architectures on given constraints.
  arXiv  Detail & Related papers  (2019-05-28T06:35:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.