$\mu$DARTS: Model Uncertainty-Aware Differentiable Architecture Search

• URL: http://arxiv.org/abs/2107.11500v1
• Date: Sat, 24 Jul 2021 01:09:20 GMT
• Title: $\mu$DARTS: Model Uncertainty-Aware Differentiable Architecture Search
• Authors: Biswadeep Chakraborty and Saibal Mukhopadhyay
• Abstract summary: We introduce concrete dropout within DARTS cells and include a Monte-Carlo regularizer within the training loss to optimize the concrete dropout probabilities. Experiments on CIFAR10, CIFAR100, SVHN, and ImageNet verify the effectiveness of $mu$DARTS in improving accuracy and reducing uncertainty.
• Score: 8.024434062411943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a Model Uncertainty-aware Differentiable ARchiTecture Search ($\mu$DARTS) that optimizes neural networks to simultaneously achieve high accuracy and low uncertainty. We introduce concrete dropout within DARTS cells and include a Monte-Carlo regularizer within the training loss to optimize the concrete dropout probabilities. A predictive variance term is introduced in the validation loss to enable searching for architecture with minimal model uncertainty. The experiments on CIFAR10, CIFAR100, SVHN, and ImageNet verify the effectiveness of $\mu$DARTS in improving accuracy and reducing uncertainty compared to existing DARTS methods. Moreover, the final architecture obtained from $\mu$DARTS shows higher robustness to noise at the input image and model parameters compared to the architecture obtained from existing DARTS methods.

Related papers

• UncertaintyRAG: Span-Level Uncertainty Enhanced Long-Context Modeling for Retrieval-Augmented Generation [93.38604803625294]
  We present UncertaintyRAG, a novel approach for long-context Retrieval-Augmented Generation (RAG) We use Signal-to-Noise Ratio (SNR)-based span uncertainty to estimate similarity between text chunks. UncertaintyRAG outperforms baselines by 2.03% on LLaMA-2-7B, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2024-10-03T17:39:38Z)
• SURE: SUrvey REcipes for building reliable and robust deep networks [12.268921703825258]
  In this paper, we revisit techniques for uncertainty estimation within deep neural networks and consolidate a suite of techniques to enhance their reliability. We rigorously evaluate SURE against the benchmark of failure prediction, a critical testbed for uncertainty estimation efficacy. When applied to real-world challenges, such as data corruption, label noise, and long-tailed class distribution, SURE exhibits remarkable robustness, delivering results that are superior or on par with current state-of-the-art specialized methods.
  arXiv  Detail & Related papers  (2024-03-01T13:58:19Z)
• Unified Probabilistic Neural Architecture and Weight Ensembling Improves Model Robustness [3.6607006319608226]
  We propose a Unified probabilistic architecture and weight ensembling Neural Architecture Search (UraeNAS) The proposed approach showed a significant improvement both with in-distribution (0.86% in accuracy) and out-of-distribution (2.43% in accuracy)
  arXiv  Detail & Related papers  (2022-10-08T18:30:30Z)
• SmoothNets: Optimizing CNN architecture design for differentially private deep learning [69.10072367807095]
  DPSGD requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. We distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training.
  arXiv  Detail & Related papers  (2022-05-09T07:51:54Z)
• ZARTS: On Zero-order Optimization for Neural Architecture Search [94.41017048659664]
  Differentiable architecture search (DARTS) has been a popular one-shot paradigm for NAS due to its high efficiency. This work turns to zero-order optimization and proposes a novel NAS scheme, called ZARTS, to search without enforcing the above approximation. In particular, results on 12 benchmarks verify the outstanding robustness of ZARTS, where the performance of DARTS collapses due to its known instability issue.
  arXiv  Detail & Related papers  (2021-10-10T09:35:15Z)
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search [11.115656548869199]
  We propose a Mean-Shift based DARTS (MS-DARTS) to improve stability based on sampling and perturbation. MS-DARTS archives higher performance over other state-of-the-art NAS methods with reduced search cost.
  arXiv  Detail & Related papers  (2021-08-23T08:06:45Z)
• D-DARTS: Distributed Differentiable Architecture Search [75.12821786565318]
  Differentiable ARchiTecture Search (DARTS) is one of the most trending Neural Architecture Search (NAS) methods. We propose D-DARTS, a novel solution that addresses this problem by nesting several neural networks at cell-level.
  arXiv  Detail & Related papers  (2021-08-20T09:07:01Z)
• Differentiable Robust LQR Layers [26.67150576276949]
  This paper proposes a differentiable robust LQR layer for reinforcement learning and imitation learning under model uncertainty and dynamics. It provides a new type of inductive bias fority and uncertainty modeling in control systems. Experiment results show that the proposed method can optimize robust policies under uncertain situations.
  arXiv  Detail & Related papers  (2021-06-10T06:52:31Z)
• DrNAS: Dirichlet Neural Architecture Search [88.56953713817545]
  We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution. With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based generalization. To alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme.
  arXiv  Detail & Related papers  (2020-06-18T08:23:02Z)
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization [99.81980366552408]
  We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. We propose a perturbation-based regularization - SmoothDARTS (SDARTS) - to smooth the loss landscape and improve the generalizability of DARTS-based methods.
  arXiv  Detail & Related papers  (2020-02-12T23:46:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.