SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search
- URL: http://arxiv.org/abs/2208.10404v1
- Date: Mon, 22 Aug 2022 15:41:28 GMT
- Title: SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search
- Authors: Zhewen Yu, Christos-Savvas Bouganis
- Abstract summary: This work proposes the SVD-NAS framework that couples the domains of low-rank approximation and neural architecture search.
Results demonstrate that the SVD-NAS achieves 2.06-12.85pp higher accuracy on ImageNet than state-of-the-art methods under the data-limited problem setting.
- Score: 7.221206118679026
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: The task of compressing pre-trained Deep Neural Networks has attracted wide
interest of the research community due to its great benefits in freeing
practitioners from data access requirements. In this domain, low-rank
approximation is a promising method, but existing solutions considered a
restricted number of design choices and failed to efficiently explore the
design space, which lead to severe accuracy degradation and limited compression
ratio achieved. To address the above limitations, this work proposes the
SVD-NAS framework that couples the domains of low-rank approximation and neural
architecture search. SVD-NAS generalises and expands the design choices of
previous works by introducing the Low-Rank architecture space, LR-space, which
is a more fine-grained design space of low-rank approximation. Afterwards, this
work proposes a gradient-descent-based search for efficiently traversing the
LR-space. This finer and more thorough exploration of the possible design
choices results in improved accuracy as well as reduction in parameters, FLOPS,
and latency of a CNN model. Results demonstrate that the SVD-NAS achieves
2.06-12.85pp higher accuracy on ImageNet than state-of-the-art methods under
the data-limited problem setting. SVD-NAS is open-sourced at
https://github.com/Yu-Zhewen/SVD-NAS.
Related papers
- Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search [5.1331676121360985]
We craft an algorithm with the capability to perform fine-grain NAS at a low cost.
We propose projecting the problem to a lower dimensional space through predicting the difference in accuracy of a pair of similar networks.
arXiv Detail & Related papers (2024-11-21T02:43:32Z) - Lightweight Monocular Depth with a Novel Neural Architecture Search
Method [46.97673710849343]
This paper presents a novel neural architecture search method, called LiDNAS, for generating lightweight monocular depth estimation models.
We construct the search space on a pre-defined backbone network to balance layer diversity and search space size.
The LiDNAS optimized models achieve results superior to compact depth estimation state-of-the-art on NYU-Depth-v2, KITTI, and ScanNet.
arXiv Detail & Related papers (2021-08-25T08:06:28Z) - Searching Efficient Model-guided Deep Network for Image Denoising [61.65776576769698]
We present a novel approach by connecting model-guided design with NAS (MoD-NAS)
MoD-NAS employs a highly reusable width search strategy and a densely connected search block to automatically select the operations of each layer.
Experimental results on several popular datasets show that our MoD-NAS has achieved even better PSNR performance than current state-of-the-art methods.
arXiv Detail & Related papers (2021-04-06T14:03:01Z) - BossNAS: Exploring Hybrid CNN-transformers with Block-wisely
Self-supervised Neural Architecture Search [100.28980854978768]
We present Block-wisely Self-supervised Neural Architecture Search (BossNAS)
We factorize the search space into blocks and utilize a novel self-supervised training scheme, named ensemble bootstrapping, to train each block separately.
We also present HyTra search space, a fabric-like hybrid CNN-transformer search space with searchable down-sampling positions.
arXiv Detail & Related papers (2021-03-23T10:05:58Z) - OPANAS: One-Shot Path Aggregation Network Architecture Search for Object
Detection [82.04372532783931]
Recently, neural architecture search (NAS) has been exploited to design feature pyramid networks (FPNs)
We propose a novel One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm, which significantly improves both searching efficiency and detection accuracy.
arXiv Detail & Related papers (2021-03-08T01:48:53Z) - Smooth Variational Graph Embeddings for Efficient Neural Architecture
Search [41.62970837629573]
We propose a two-sided variational graph autoencoder, which allows to smoothly encode and accurately reconstruct neural architectures from various search spaces.
We evaluate the proposed approach on neural architectures defined by the ENAS approach, the NAS-Bench-101 and the NAS-Bench-201 search spaces.
arXiv Detail & Related papers (2020-10-09T17:05:41Z) - Binarized Neural Architecture Search for Efficient Object Recognition [120.23378346337311]
Binarized neural architecture search (BNAS) produces extremely compressed models to reduce huge computational cost on embedded devices for edge computing.
An accuracy of $96.53%$ vs. $97.22%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40%$ faster search than the state-of-the-art PC-DARTS.
arXiv Detail & Related papers (2020-09-08T15:51:23Z) - DrNAS: Dirichlet Neural Architecture Search [88.56953713817545]
We treat the continuously relaxed architecture mixing weight as random variables, modeled by Dirichlet distribution.
With recently developed pathwise derivatives, the Dirichlet parameters can be easily optimized with gradient-based generalization.
To alleviate the large memory consumption of differentiable NAS, we propose a simple yet effective progressive learning scheme.
arXiv Detail & Related papers (2020-06-18T08:23:02Z) - Geometry-Aware Gradient Algorithms for Neural Architecture Search [41.943045315986744]
We argue for the study of single-level empirical risk minimization to understand NAS with weight-sharing.
We present a geometry-aware framework that exploits the underlying structure of this optimization to return sparse architectural parameters.
We achieve state-of-the-art accuracy on the latest NAS benchmarks in computer vision.
arXiv Detail & Related papers (2020-04-16T17:46:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.