Multi-objective Differentiable Neural Architecture Search

• URL: http://arxiv.org/abs/2402.18213v2
• Date: Wed, 19 Jun 2024 12:15:20 GMT
• Title: Multi-objective Differentiable Neural Architecture Search
• Authors: Rhea Sanjay Sukthanker, Arber Zela, Benedikt Staffler, Samuel Dooley, Josif Grabocka, Frank Hutter,
• Abstract summary: We propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics. Our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets.
• Score: 58.67218773054753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training. Typically, in MOO neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a computationally expensive search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics, and yields representative and diverse architectures across multiple devices in just one search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments with up to 19 hardware devices and 3 objectives showcase the effectiveness and scalability of our method. Finally, we show that, without extra costs, our method outperforms existing MOO NAS methods across a broad range of qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k, an encoder-decoder transformer space for machine translation and a decoder-only transformer space for language modelling.

Related papers

• Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity [12.52012450501367]
  Hardware-aware Neural Architecture Search approaches (HW-NAS) automate the design of deep learning architectures tailored specifically to a given target hardware platform. These techniques demand substantial computational resources, primarily due to the expensive process of assessing the performance of identified architectures. We propose a Multi-Objective method to address the HW-NAS problem, called MO-HDNAS, to identify the trade-off set of architectures in a single run with low computational cost.
  arXiv  Detail & Related papers  (2024-04-15T15:32:58Z)
• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• Search-time Efficient Device Constraints-Aware Neural Architecture Search [6.527454079441765]
  Deep learning techniques like computer vision and natural language processing can be computationally expensive and memory-intensive. We automate the construction of task-specific deep learning architectures optimized for device constraints through Neural Architecture Search (NAS) We present DCA-NAS, a principled method of fast neural network architecture search that incorporates edge-device constraints.
  arXiv  Detail & Related papers  (2023-07-10T09:52:28Z)
• OFA$^2$: A Multi-Objective Perspective for the Once-for-All Neural Architecture Search [79.36688444492405]
  Once-for-All (OFA) is a Neural Architecture Search (NAS) framework designed to address the problem of searching efficient architectures for devices with different resources constraints. We aim to give one step further in the search for efficiency by explicitly conceiving the search stage as a multi-objective optimization problem.
  arXiv  Detail & Related papers  (2023-03-23T21:30:29Z)
• Lightweight Neural Architecture Search for Temporal Convolutional Networks at the Edge [21.72253397805102]
  This work focuses in particular on Temporal Convolutional Networks (TCNs), a convolutional model for time-series processing. We propose the first NAS tool that explicitly targets the optimization of the most peculiar architectural parameters of TCNs. We test the proposed NAS on four real-world, edge-relevant tasks, involving audio and bio-signals.
  arXiv  Detail & Related papers  (2023-01-24T19:47:40Z)
• You Only Search Once: On Lightweight Differentiable Architecture Search for Resource-Constrained Embedded Platforms [10.11289927237036]
  Differentiable neural architecture search (NAS) has evolved as the most dominant alternative to automatically design competitive deep neural networks (DNNs) We introduce a lightweight hardware-aware differentiable NAS framework dubbed LightNAS, striving to find the required architecture through a one-time search. Extensive experiments are conducted to show the superiority of LightNAS over previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-08-30T02:23:23Z)
• Searching for Efficient Neural Architectures for On-Device ML on Edge TPUs [10.680700357879601]
  Neural architecture search (NAS) comes to the rescue for efficiently utilizing the high compute throughput offered by on-device ML accelerators. Existing NAS frameworks have several practical limitations in scaling to multiple tasks and different target platforms. We provide a two-pronged approach to this challenge: (i) a neural architecture that decouples model cost evaluation, search space design, and the algorithm to rapidly target various on-device ML tasks, and (ii) search spaces crafted from group convolution based inverted bottleneck (IBN) variants.
  arXiv  Detail & Related papers  (2022-04-09T00:35:19Z)
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets [93.79297053429447]
  Existing methods often perform an independent architecture search for each target budget. We propose a general architecture generator that automatically produces effective architectures for an arbitrary budget merely via model inference. Extensive experiments on three platforms (i.e., mobile, CPU, and GPU) show the superiority of the proposed method over existing NAS methods.
  arXiv  Detail & Related papers  (2021-02-27T13:59:17Z)
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning [71.90902837008278]
  We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL) In order to adapt to different task combinations, we disentangle the GP-MTL networks into single-task backbones. We also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures.
  arXiv  Detail & Related papers  (2020-03-31T09:49:14Z)
• NAS-Count: Counting-by-Density with Neural Architecture Search [74.92941571724525]
  We automate the design of counting models with Neural Architecture Search (NAS) We introduce an end-to-end searched encoder-decoder architecture, Automatic Multi-Scale Network (AMSNet)
  arXiv  Detail & Related papers  (2020-02-29T09:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.