A Design Space Study for LISTA and Beyond
- URL: http://arxiv.org/abs/2104.04110v1
- Date: Thu, 8 Apr 2021 23:01:52 GMT
- Title: A Design Space Study for LISTA and Beyond
- Authors: Tianjian Meng, Xiaohan Chen, Yifan Jiang, Zhangyang Wang
- Abstract summary: In recent years, great success has been witnessed in building problem-specific deep networks from unrolling iterative algorithms.
This paper revisits the role of unrolling as a design approach for deep networks, to what extent its resulting special architecture is superior, and can we find better?
Using LISTA for sparse recovery as a representative example, we conduct the first thorough design space study for the unrolled models.
- Score: 79.76740811464597
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In recent years, great success has been witnessed in building
problem-specific deep networks from unrolling iterative algorithms, for solving
inverse problems and beyond. Unrolling is believed to incorporate the
model-based prior with the learning capacity of deep learning. This paper
revisits the role of unrolling as a design approach for deep networks: to what
extent its resulting special architecture is superior, and can we find better?
Using LISTA for sparse recovery as a representative example, we conduct the
first thorough design space study for the unrolled models. Among all possible
variations, we focus on extensively varying the connectivity patterns and
neuron types, leading to a gigantic design space arising from LISTA. To
efficiently explore this space and identify top performers, we leverage the
emerging tool of neural architecture search (NAS). We carefully examine the
searched top architectures in a number of settings, and are able to discover
networks that are consistently better than LISTA. We further present more
visualization and analysis to "open the black box", and find that the searched
top architectures demonstrate highly consistent and potentially transferable
patterns. We hope our study to spark more reflections and explorations on how
to better mingle model-based optimization prior and data-driven learning.
Related papers
- Visual Prompting Upgrades Neural Network Sparsification: A Data-Model
Perspective [67.25782152459851]
We introduce a novel data-model co-design perspective: to promote superior weight sparsity.
Specifically, customized Visual Prompts are mounted to upgrade neural Network sparsification in our proposed VPNs framework.
arXiv Detail & Related papers (2023-12-03T13:50:24Z) - Masked Autoencoders Are Robust Neural Architecture Search Learners [14.965550562292476]
We propose a novel NAS framework based on Masked Autoencoders (MAE) that eliminates the need for labeled data during the search process.
By replacing the supervised learning objective with an image reconstruction task, our approach enables the robust discovery of network architectures.
arXiv Detail & Related papers (2023-11-20T13:45:21Z) - Searching a High-Performance Feature Extractor for Text Recognition
Network [92.12492627169108]
We design a domain-specific search space by exploring principles for having good feature extractors.
As the space is huge and complexly structured, no existing NAS algorithms can be applied.
We propose a two-stage algorithm to effectively search in the space.
arXiv Detail & Related papers (2022-09-27T03:49:04Z) - SuperNet in Neural Architecture Search: A Taxonomic Survey [14.037182039950505]
This survey focuses on the supernet optimization that builds a neural network that assembles all the architectures as its sub models by using weight sharing.
We aim to accomplish that by proposing them as solutions to the common challenges found in the literature: data-side optimization, poor rank correlation alleviation, and transferable NAS for a number of deployment scenarios.
arXiv Detail & Related papers (2022-04-08T08:29:52Z) - One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search
Space Shrinking [97.60915598958968]
We propose a one-shot neural ensemble architecture search (NEAS) solution that addresses the two challenges.
For the first challenge, we introduce a novel diversity-based metric to guide search space shrinking.
For the second challenge, we enable a new search dimension to learn layer sharing among different models for efficiency purposes.
arXiv Detail & Related papers (2021-04-01T16:29:49Z) - NAS-DIP: Learning Deep Image Prior with Neural Architecture Search [65.79109790446257]
Recent work has shown that the structure of deep convolutional neural networks can be used as a structured image prior.
We propose to search for neural architectures that capture stronger image priors.
We search for an improved network by leveraging an existing neural architecture search algorithm.
arXiv Detail & Related papers (2020-08-26T17:59:36Z) - AlphaGAN: Fully Differentiable Architecture Search for Generative
Adversarial Networks [15.740179244963116]
Generative Adversarial Networks (GANs) are formulated as minimax game problems, whereby generators attempt to approach real data distributions by virtue of adversarial learning against discriminators.
In this work, we aim to boost model learning from the perspective of network architectures, by incorporating recent progress on automated architecture search into GANs.
We propose a fully differentiable search framework for generative adversarial networks, dubbed alphaGAN.
arXiv Detail & Related papers (2020-06-16T13:27:30Z) - Stage-Wise Neural Architecture Search [65.03109178056937]
Modern convolutional networks such as ResNet and NASNet have achieved state-of-the-art results in many computer vision applications.
These networks consist of stages, which are sets of layers that operate on representations in the same resolution.
It has been demonstrated that increasing the number of layers in each stage improves the prediction ability of the network.
However, the resulting architecture becomes computationally expensive in terms of floating point operations, memory requirements and inference time.
arXiv Detail & Related papers (2020-04-23T14:16:39Z) - Neural Architecture Generator Optimization [9.082931889304723]
We are first to investigate casting NAS as a problem of finding the optimal network generator.
We propose a new, hierarchical and graph-based search space capable of representing an extremely large variety of network types.
arXiv Detail & Related papers (2020-04-03T06:38:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.