RepNAS: Searching for Efficient Re-parameterizing Blocks
- URL: http://arxiv.org/abs/2109.03508v1
- Date: Wed, 8 Sep 2021 09:04:59 GMT
- Title: RepNAS: Searching for Efficient Re-parameterizing Blocks
- Authors: Mingyang Zhang, Xinyi Yu, Jingtao Rong, Linlin Ou, Feng Gao
- Abstract summary: RepNAS, a one-stage NAS approach, is present to efficiently search the optimal diverse branch block(ODBB) for each layer under the branch number constraint.
Our experimental results show the searched ODBB can easily surpass the manual diverse branch block(DBB) with efficient training.
- Score: 4.146471448631912
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In the past years, significant improvements in the field of neural
architecture search(NAS) have been made. However, it is still challenging to
search for efficient networks due to the gap between the searched constraint
and real inference time exists. To search for a high-performance network with
low inference time, several previous works set a computational complexity
constraint for the search algorithm. However, many factors affect the speed of
inference(e.g., FLOPs, MACs). The correlation between a single indicator and
the latency is not strong. Currently, some re-parameterization(Rep) techniques
are proposed to convert multi-branch to single-path architecture which is
inference-friendly. Nevertheless, multi-branch architectures are still
human-defined and inefficient. In this work, we propose a new search space that
is suitable for structural re-parameterization techniques. RepNAS, a one-stage
NAS approach, is present to efficiently search the optimal diverse branch
block(ODBB) for each layer under the branch number constraint. Our experimental
results show the searched ODBB can easily surpass the manual diverse branch
block(DBB) with efficient training. Code and models will be available sooner.
Related papers
- Lightweight Diffusion Models with Distillation-Based Block Neural
Architecture Search [55.41583104734349]
We propose to automatically remove structural redundancy in diffusion models with our proposed Diffusion Distillation-based Block-wise Neural Architecture Search (NAS)
Given a larger pretrained teacher, we leverage DiffNAS to search for the smallest architecture which can achieve on-par or even better performance than the teacher.
Different from previous block-wise NAS methods, DiffNAS contains a block-wise local search strategy and a retraining strategy with a joint dynamic loss.
arXiv Detail & Related papers (2023-11-08T12:56:59Z) - $\beta$-DARTS: Beta-Decay Regularization for Differentiable Architecture
Search [85.84110365657455]
We propose a simple-but-efficient regularization method, termed as Beta-Decay, to regularize the DARTS-based NAS searching process.
Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets.
arXiv Detail & Related papers (2022-03-03T11:47:14Z) - Zero-Cost Proxies Meet Differentiable Architecture Search [20.957570100784988]
Differentiable neural architecture search (NAS) has attracted significant attention in recent years.
Despite its success, DARTS lacks robustness in certain cases.
We propose a novel operation selection paradigm in the context of differentiable NAS.
arXiv Detail & Related papers (2021-06-12T15:33:36Z) - Making Differentiable Architecture Search less local [9.869449181400466]
Differentiable neural architecture search (DARTS) is a promising NAS approach that dramatically increases search efficiency.
It has been shown to suffer from performance collapse, where the search often leads to detrimental architectures.
We develop a more global optimisation scheme that is able to better explore the space without changing the DARTS problem formulation.
arXiv Detail & Related papers (2021-04-21T10:36:43Z) - BossNAS: Exploring Hybrid CNN-transformers with Block-wisely
Self-supervised Neural Architecture Search [100.28980854978768]
We present Block-wisely Self-supervised Neural Architecture Search (BossNAS)
We factorize the search space into blocks and utilize a novel self-supervised training scheme, named ensemble bootstrapping, to train each block separately.
We also present HyTra search space, a fabric-like hybrid CNN-transformer search space with searchable down-sampling positions.
arXiv Detail & Related papers (2021-03-23T10:05:58Z) - Towards Improving the Consistency, Efficiency, and Flexibility of
Differentiable Neural Architecture Search [84.4140192638394]
Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation.
In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells.
Our method also spares much memory and computation cost, which speeds up the search process.
arXiv Detail & Related papers (2021-01-27T12:16:47Z) - Neural Architecture Search via Combinatorial Multi-Armed Bandit [43.29214413461234]
We formulate NAS as a Combinatorial Multi-Armed Bandit (CMAB) problem (CMAB-NAS)
This allows the decomposition of a large search space into smaller blocks where tree-search methods can be applied more effectively and efficiently.
We leverage a tree-based method called Nested Monte-Carlo Search to tackle the CMAB-NAS problem.
On CIFAR-10, our approach discovers a cell structure that achieves a low error rate that is comparable to the state-of-the-art, using only 0.58 GPU days.
arXiv Detail & Related papers (2021-01-01T23:29:33Z) - ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse
Coding [86.40042104698792]
We formulate neural architecture search as a sparse coding problem.
In experiments, our two-stage method on CIFAR-10 requires only 0.05 GPU-day for search.
Our one-stage method produces state-of-the-art performances on both CIFAR-10 and ImageNet at the cost of only evaluation time.
arXiv Detail & Related papers (2020-10-13T04:34:24Z) - CATCH: Context-based Meta Reinforcement Learning for Transferrable
Architecture Search [102.67142711824748]
CATCH is a novel Context-bAsed meTa reinforcement learning algorithm for transferrable arChitecture searcH.
The combination of meta-learning and RL allows CATCH to efficiently adapt to new tasks while being agnostic to search spaces.
It is also capable of handling cross-domain architecture search as competitive networks on ImageNet, COCO, and Cityscapes are identified.
arXiv Detail & Related papers (2020-07-18T09:35:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.