FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness

• URL: http://arxiv.org/abs/2402.19102v1
• Date: Thu, 29 Feb 2024 12:33:14 GMT
• Title: FlatNAS: optimizing Flatness in Neural Architecture Search for Out-of-Distribution Robustness
• Authors: Matteo Gambella, Fabrizio Pittorino, and Manuel Roveri
• Abstract summary: This study introduces a novel NAS solution, called Flat Neural Architecture Search (FlatNAS) It explores the interplay between a novel figure of merit based on robustness to weight perturbations and single NN optimization with Sharpness-Aware Minimization (SAM) The OOD robustness of the NAS-designed models is evaluated by focusing on robustness to input data corruptions, using popular benchmark datasets in the literature.
• Score: 3.724847012963521
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Architecture Search (NAS) paves the way for the automatic definition of Neural Network (NN) architectures, attracting increasing research attention and offering solutions in various scenarios. This study introduces a novel NAS solution, called Flat Neural Architecture Search (FlatNAS), which explores the interplay between a novel figure of merit based on robustness to weight perturbations and single NN optimization with Sharpness-Aware Minimization (SAM). FlatNAS is the first work in the literature to systematically explore flat regions in the loss landscape of NNs in a NAS procedure, while jointly optimizing their performance on in-distribution data, their out-of-distribution (OOD) robustness, and constraining the number of parameters in their architecture. Differently from current studies primarily concentrating on OOD algorithms, FlatNAS successfully evaluates the impact of NN architectures on OOD robustness, a crucial aspect in real-world applications of machine and deep learning. FlatNAS achieves a good trade-off between performance, OOD generalization, and the number of parameters, by using only in-distribution data in the NAS exploration. The OOD robustness of the NAS-designed models is evaluated by focusing on robustness to input data corruptions, using popular benchmark datasets in the literature.

Related papers

• Fair Differentiable Neural Network Architecture Search for Long-Tailed Data with Self-Supervised Learning [0.0]
  This paper explores to improve the searching and training performance of NAS on long-tailed datasets. We first discuss the related works about NAS and the deep learning method for long-tailed datasets. Then, we focus on an existing work, called SSF-NAS, which integrates the self-supervised learning and fair differentiable NAS. Finally, we conducted a series of experiments on the CIFAR10-LT dataset for performance evaluation.
  arXiv  Detail & Related papers  (2024-06-19T12:39:02Z)
• Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations [59.683234126055694]
  Recent neural architecture search (NAS) frameworks have been successful in finding optimal architectures for given conditions. We propose a novel lightweight robust zero-cost proxy that considers the consistency across features, parameters, and gradients of both clean and perturbed images. Our approach facilitates an efficient and rapid search for neural architectures capable of learning generalizable features that exhibit robustness across diverse perturbations.
  arXiv  Detail & Related papers  (2023-06-08T08:34:26Z)
• Robust Neural Architecture Search [19.214462477848535]
  We propose a novel NAS method, Robust Neural Architecture Search (RNAS) To design a regularization term to balance accuracy and robustness, RNAS generates architectures with both high accuracy and good robustness. Experiments show that RNAS achieves state-of-the-art (SOTA) performance on both image classification and adversarial attacks.
  arXiv  Detail & Related papers  (2023-04-06T03:21:24Z)
• Generalization Properties of NAS under Activation and Skip Connection Search [66.8386847112332]
  We study the generalization properties of Neural Architecture Search (NAS) under a unifying framework. We derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime. We show how the derived results can guide NAS to select the top-performing architectures, even in the case without training.
  arXiv  Detail & Related papers  (2022-09-15T12:11:41Z)
• BaLeNAS: Differentiable Architecture Search via the Bayesian Learning Rule [95.56873042777316]
  Differentiable Architecture Search (DARTS) has received massive attention in recent years, mainly because it significantly reduces the computational cost. This paper formulates the neural architecture search as a distribution learning problem through relaxing the architecture weights into Gaussian distributions. We demonstrate how the differentiable NAS benefits from Bayesian principles, enhancing exploration and improving stability.
  arXiv  Detail & Related papers  (2021-11-25T18:13:42Z)
• Lightweight Monocular Depth with a Novel Neural Architecture Search Method [46.97673710849343]
  This paper presents a novel neural architecture search method, called LiDNAS, for generating lightweight monocular depth estimation models. We construct the search space on a pre-defined backbone network to balance layer diversity and search space size. The LiDNAS optimized models achieve results superior to compact depth estimation state-of-the-art on NYU-Depth-v2, KITTI, and ScanNet.
  arXiv  Detail & Related papers  (2021-08-25T08:06:28Z)
• Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
  We present a novel approach by connecting model-guided design with NAS (MoD-NAS) MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer. Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-04-06T14:03:01Z)
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment [23.988393741948485]
  We propose a novel search strategy for one-shot and sparse propagation NAS, namely AdvantageNAS. AdvantageNAS is a gradient-based approach that improves the search efficiency by introducing credit assignment in gradient estimation for architecture updates. Experiments on the NAS-Bench-201 and PTB dataset show that AdvantageNAS discovers an architecture with higher performance under a limited time budget.
  arXiv  Detail & Related papers  (2020-12-11T05:45:03Z)
• Neural Network Design: Learning from Neural Architecture Search [3.9430294028981763]
  Neural Architecture Search (NAS) aims to optimize deep neural networks' architecture for better accuracy or smaller computational cost. Despite various successful approaches proposed to solve the NAS task, the landscape of it, along with its properties, are rarely investigated.
  arXiv  Detail & Related papers  (2020-11-01T15:02:02Z)
• Binarized Neural Architecture Search for Efficient Object Recognition [120.23378346337311]
  Binarized neural architecture search (BNAS) produces extremely compressed models to reduce huge computational cost on embedded devices for edge computing. An accuracy of $96.53%$ vs. $97.22%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40%$ faster search than the state-of-the-art PC-DARTS.
  arXiv  Detail & Related papers  (2020-09-08T15:51:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.