FL-NAS: Towards Fairness of NAS for Resource Constrained Devices via Large Language Models

• URL: http://arxiv.org/abs/2402.06696v1
• Date: Fri, 9 Feb 2024 00:49:03 GMT
• Title: FL-NAS: Towards Fairness of NAS for Resource Constrained Devices via Large Language Models
• Authors: Ruiyang Qin, Yuting Hu, Zheyu Yan, Jinjun Xiong, Ahmed Abbasi, Yiyu Shi
• Abstract summary: This paper conducts further exploration in this direction by considering three important design metrics simultaneously. We propose a novel LLM-based NAS framework, FL-NAS, in this paper. We show experimentally that FL-NAS can indeed find high-performing DNNs, beating state-of-the-art DNN models by orders-of-magnitude across almost all design considerations.
• Score: 24.990028167518226
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Architecture Search (NAS) has become the de fecto tools in the industry in automating the design of deep neural networks for various applications, especially those driven by mobile and edge devices with limited computing resources. The emerging large language models (LLMs), due to their prowess, have also been incorporated into NAS recently and show some promising results. This paper conducts further exploration in this direction by considering three important design metrics simultaneously, i.e., model accuracy, fairness, and hardware deployment efficiency. We propose a novel LLM-based NAS framework, FL-NAS, in this paper, and show experimentally that FL-NAS can indeed find high-performing DNNs, beating state-of-the-art DNN models by orders-of-magnitude across almost all design considerations.

Related papers

• Fair Differentiable Neural Network Architecture Search for Long-Tailed Data with Self-Supervised Learning [0.0]
  This paper explores to improve the searching and training performance of NAS on long-tailed datasets. We first discuss the related works about NAS and the deep learning method for long-tailed datasets. Then, we focus on an existing work, called SSF-NAS, which integrates the self-supervised learning and fair differentiable NAS. Finally, we conducted a series of experiments on the CIFAR10-LT dataset for performance evaluation.
  arXiv  Detail & Related papers  (2024-06-19T12:39:02Z)
• FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness [3.724847012963521]
  This study introduces a novel NAS solution, called Flat Neural Architecture Search (FlatNAS) It explores the interplay between a novel figure of merit based on robustness to weight perturbations and single NN optimization with Sharpness-Aware Minimization (SAM) The OOD robustness of the NAS-designed models is evaluated by focusing on robustness to input data corruptions, using popular benchmark datasets in the literature.
  arXiv  Detail & Related papers  (2024-02-29T12:33:14Z)
• DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
  We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG. Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them. We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS. When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
  arXiv  Detail & Related papers  (2023-05-26T13:58:18Z)
• Data Aware Neural Architecture Search [0.12891210250935145]
  In Machine Learning, one single metric is not enough to evaluate a NN architecture. Recent works on NAS for resource constrained systems have investigated various approaches to optimize for multiple metrics. We name such a system "Data Aware NAS", and we provide experimental evidence of its benefits.
  arXiv  Detail & Related papers  (2023-04-04T14:20:36Z)
• Generalization Properties of NAS under Activation and Skip Connection Search [66.8386847112332]
  We study the generalization properties of Neural Architecture Search (NAS) under a unifying framework. We derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime. We show how the derived results can guide NAS to select the top-performing architectures, even in the case without training.
  arXiv  Detail & Related papers  (2022-09-15T12:11:41Z)
• U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search [50.33956216274694]
  optimizing resource utilization in target platforms is key to achieving high performance during DNN inference. We propose a novel hardware-aware NAS framework that does not only optimize for task accuracy and inference latency, but also for resource utilization. We achieve 2.8 - 4x speedup for DNN inference compared to prior hardware-aware NAS methods.
  arXiv  Detail & Related papers  (2022-03-23T13:44:15Z)
• FLASH: Fast Neural Architecture Search with Hardware Optimization [7.263481020106725]
  Neural architecture search (NAS) is a promising technique to design efficient and high-performance deep neural networks (DNNs) This paper proposes FLASH, a very fast NAS methodology that co-optimizes the DNN accuracy and performance on a real hardware platform.
  arXiv  Detail & Related papers  (2021-08-01T23:46:48Z)
• Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
  We present a novel approach by connecting model-guided design with NAS (MoD-NAS) MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer. Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-06T14:03:01Z)
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers [18.662026553041937]
  Machine learning on resource constrained microcontrollers (MCUs) promises to drastically expand the application space of the Internet of Things (IoT) TinyML presents severe technical challenges, as deep neural network inference demands a large compute and memory budget. neural architecture search (NAS) promises to help design accurate ML models that meet the tight MCU memory, latency and energy constraints.
  arXiv  Detail & Related papers  (2020-10-21T19:39:39Z)
• MS-RANAS: Multi-Scale Resource-Aware Neural Architecture Search [94.80212602202518]
  We propose Multi-Scale Resource-Aware Neural Architecture Search (MS-RANAS) We employ a one-shot architecture search approach in order to obtain a reduced search cost. We achieve state-of-the-art results in terms of accuracy-speed trade-off.
  arXiv  Detail & Related papers  (2020-09-29T11:56:01Z)
• Binarized Neural Architecture Search for Efficient Object Recognition [120.23378346337311]
  Binarized neural architecture search (BNAS) produces extremely compressed models to reduce huge computational cost on embedded devices for edge computing. An accuracy of $96.53%$ vs. $97.22%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40%$ faster search than the state-of-the-art PC-DARTS.
  arXiv  Detail & Related papers  (2020-09-08T15:51:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.