Proxyless Neural Architecture Adaptation for Supervised Learning and Self-Supervised Learning

• URL: http://arxiv.org/abs/2205.07168v1
• Date: Sun, 15 May 2022 02:49:48 GMT
• Title: Proxyless Neural Architecture Adaptation for Supervised Learning and Self-Supervised Learning
• Authors: Do-Guk Kim, Heung-Chang Lee
• Abstract summary: We propose proxyless neural architecture adaptation that is reproducible and efficient. Our method can be applied to both supervised learning and self-supervised learning.
• Score: 3.766702945560518
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Neural Architecture Search (NAS) methods have been introduced and show impressive performance on many benchmarks. Among those NAS studies, Neural Architecture Transformer (NAT) aims to adapt the given neural architecture to improve performance while maintaining computational costs. However, NAT lacks reproducibility and it requires an additional architecture adaptation process before network weight training. In this paper, we propose proxyless neural architecture adaptation that is reproducible and efficient. Our method can be applied to both supervised learning and self-supervised learning. The proposed method shows stable performance on various architectures. Extensive reproducibility experiments on two datasets, i.e., CIFAR-10 and Tiny Imagenet, present that the proposed method definitely outperforms NAT and is applicable to other models and datasets.

Related papers

• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• DCP-NAS: Discrepant Child-Parent Neural Architecture Search for 1-bit CNNs [53.82853297675979]
  1-bit convolutional neural networks (CNNs) with binary weights and activations show their potential for resource-limited embedded devices. One natural approach is to use 1-bit CNNs to reduce the computation and memory cost of NAS. We introduce Discrepant Child-Parent Neural Architecture Search (DCP-NAS) to efficiently search 1-bit CNNs.
  arXiv  Detail & Related papers  (2023-06-27T11:28:29Z)
• Training-free Neural Architecture Search for RNNs and Transformers [0.0]
  We develop a new training-free metric, named hidden covariance, that predicts the trained performance of an RNN architecture. We find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search.
  arXiv  Detail & Related papers  (2023-06-01T02:06:13Z)
• NAR-Former: Neural Architecture Representation Learning towards Holistic Attributes Prediction [37.357949900603295]
  We propose a neural architecture representation model that can be used to estimate attributes holistically. Experiment results show that our proposed framework can be used to predict the latency and accuracy attributes of both cell architectures and whole deep neural networks.
  arXiv  Detail & Related papers  (2022-11-15T10:15:21Z)
• Neural Architecture Search for Speech Emotion Recognition [72.1966266171951]
  We propose to apply neural architecture search (NAS) techniques to automatically configure the SER models. We show that NAS can improve SER performance (54.89% to 56.28%) while maintaining model parameter sizes.
  arXiv  Detail & Related papers  (2022-03-31T10:16:10Z)
• Learning Interpretable Models Through Multi-Objective Neural Architecture Search [0.9990687944474739]
  We propose a framework to optimize for both task performance and "introspectability," a surrogate metric for aspects of interpretability. We demonstrate that jointly optimizing for task error and introspectability leads to more disentangled and debuggable architectures that perform within error.
  arXiv  Detail & Related papers  (2021-12-16T05:50:55Z)
• Self-Denoising Neural Networks for Few Shot Learning [66.38505903102373]
  We present a new training scheme that adds noise at multiple stages of an existing neural architecture while simultaneously learning to be robust to this added noise. This architecture, which we call a Self-Denoising Neural Network (SDNN), can be applied easily to most modern convolutional neural architectures.
  arXiv  Detail & Related papers  (2021-10-26T03:28:36Z)
• Conceptual Expansion Neural Architecture Search (CENAS) [1.3464152928754485]
  We present an approach called Conceptual Expansion Neural Architecture Search (CENAS) It combines a sample-efficient, computational creativity-inspired transfer learning approach with neural architecture search. It finds models faster than naive architecture search via transferring existing weights to approximate the parameters of the new model.
  arXiv  Detail & Related papers  (2021-10-07T02:29:26Z)
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search [65.79109790446257]
  Recent work has shown that the structure of deep convolutional neural networks can be used as a structured image prior. We propose to search for neural architectures that capture stronger image priors. We search for an improved network by leveraging an existing neural architecture search algorithm.
  arXiv  Detail & Related papers  (2020-08-26T17:59:36Z)
• Differentiable Neural Architecture Transformation for Reproducible Architecture Improvement [3.766702945560518]
  We propose differentiable neural architecture transformation that is reproducible and efficient. Extensive experiments on two datasets, i.e., CIFAR-10 and Tiny Imagenet, present that the proposed method definitely outperforms NAT.
  arXiv  Detail & Related papers  (2020-06-15T09:03:48Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.