Enhancing the Robustness, Efficiency, and Diversity of Differentiable
Architecture Search
- URL: http://arxiv.org/abs/2204.04681v1
- Date: Sun, 10 Apr 2022 13:25:36 GMT
- Title: Enhancing the Robustness, Efficiency, and Diversity of Differentiable
Architecture Search
- Authors: Chao Li, Jia Ning, Han Hu, Kun He
- Abstract summary: Differentiable architecture search (DARTS) has attracted much attention due to its simplicity and significant improvement in efficiency.
Many works attempt to restrict the accumulation of skip connections by indicators or manual design.
We suggest a more subtle and direct approach that removes skip connections from the operation space.
- Score: 25.112048502327738
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Differentiable architecture search (DARTS) has attracted much attention due
to its simplicity and significant improvement in efficiency. However, the
excessive accumulation of the skip connection makes it suffer from long-term
weak stability and low robustness. Many works attempt to restrict the
accumulation of skip connections by indicators or manual design, however, these
methods are susceptible to thresholds and human priors. In this work, we
suggest a more subtle and direct approach that removes skip connections from
the operation space. Then, by introducing an adaptive channel allocation
strategy, we redesign the DARTS framework to automatically refill the skip
connections in the evaluation stage, resolving the performance degradation
caused by the absence of skip connections. Our method, dubbed
Adaptive-Channel-Allocation-DARTS (ACA-DRATS), could eliminate the
inconsistency in operation strength and significantly expand the architecture
diversity. We continue to explore smaller search space under our framework, and
offer a direct search on the entire ImageNet dataset. Experiments show that
ACA-DRATS improves the search stability and significantly speeds up DARTS by
more than ten times while yielding higher accuracy.
Related papers
- Adaptive Anomaly Detection in Network Flows with Low-Rank Tensor Decompositions and Deep Unrolling [9.20186865054847]
Anomaly detection (AD) is increasingly recognized as a key component for ensuring the resilience of future communication systems.
This work considers AD in network flows using incomplete measurements.
We propose a novel block-successive convex approximation algorithm based on a regularized model-fitting objective.
Inspired by Bayesian approaches, we extend the model architecture to perform online adaptation to per-flow and per-time-step statistics.
arXiv Detail & Related papers (2024-09-17T19:59:57Z) - Efficient Architecture Search via Bi-level Data Pruning [70.29970746807882]
This work pioneers an exploration into the critical role of dataset characteristics for DARTS bi-level optimization.
We introduce a new progressive data pruning strategy that utilizes supernet prediction dynamics as the metric.
Comprehensive evaluations on the NAS-Bench-201 search space, DARTS search space, and MobileNet-like search space validate that BDP reduces search costs by over 50%.
arXiv Detail & Related papers (2023-12-21T02:48:44Z) - Robustifying DARTS by Eliminating Information Bypass Leakage via
Explicit Sparse Regularization [8.93957397187611]
Differentiable architecture search (DARTS) is a promising end to end NAS method.
Recent studies cast doubt on the basic underlying hypotheses of DARTS.
We propose a novel sparse-regularized approximation and an efficient mixed-sparsity training scheme to robustify DARTS.
arXiv Detail & Related papers (2023-06-12T04:11:37Z) - Operation-level Progressive Differentiable Architecture Search [19.214462477848535]
We propose operation-level progressive differentiable neural architecture search (OPP-DARTS) to avoid skip connections aggregation.
Our method's performance on CIFAR-10 is superior to the architecture found by standard DARTS.
arXiv Detail & Related papers (2023-02-11T09:18:01Z) - Partial Connection Based on Channel Attention for Differentiable Neural
Architecture Search [1.1125818448814198]
Differentiable neural architecture search (DARTS) is a gradient-guided search method.
The parameters of some weight-equipped operations may not be trained well in the initial stage.
A partial channel connection based on channel attention for differentiable neural architecture search (ADARTS) is proposed.
arXiv Detail & Related papers (2022-08-01T12:05:55Z) - $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture
Search [85.84110365657455]
We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process.
Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
arXiv Detail & Related papers (2022-03-03T11:47:14Z) - CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference.
We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms.
Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
arXiv Detail & Related papers (2021-10-21T06:26:31Z) - Learning to Perform Downlink Channel Estimation in Massive MIMO Systems [72.76968022465469]
We study downlink (DL) channel estimation in a Massive multiple-input multiple-output (MIMO) system.
A common approach is to use the mean value as the estimate, motivated by channel hardening.
We propose two novel estimation methods.
arXiv Detail & Related papers (2021-09-06T13:42:32Z) - Decoupled and Memory-Reinforced Networks: Towards Effective Feature
Learning for One-Step Person Search [65.51181219410763]
One-step methods have been developed to handle pedestrian detection and identification sub-tasks using a single network.
There are two major challenges in the current one-step approaches.
We propose a decoupled and memory-reinforced network (DMRNet) to overcome these problems.
arXiv Detail & Related papers (2021-02-22T06:19:45Z) - Theory-Inspired Path-Regularized Differential Network Architecture
Search [206.93821077400733]
We study the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in differential architecture search (DARTS)
We propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that converge slower than shallow ones.
arXiv Detail & Related papers (2020-06-30T05:28:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.