Contrastive Embeddings for Neural Architectures
- URL: http://arxiv.org/abs/2102.04208v1
- Date: Mon, 8 Feb 2021 14:06:35 GMT
- Title: Contrastive Embeddings for Neural Architectures
- Authors: Daniel Hesslow and Iacopo Poli
- Abstract summary: We show that traditional black-box optimization algorithms, without modification, can reach state-of-the-art performance in Neural Architecture Search.
We also show the evolution of embeddings during training, motivating future studies into using embeddings at different training stages to gain a deeper understanding of the networks in a search space.
- Score: 1.90365714903665
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The performance of algorithms for neural architecture search strongly depends
on the parametrization of the search space. We use contrastive learning to
identify networks across different initializations based on their data
Jacobians, and automatically produce the first architecture embeddings
independent from the parametrization of the search space. Using our contrastive
embeddings, we show that traditional black-box optimization algorithms, without
modification, can reach state-of-the-art performance in Neural Architecture
Search. As our method provides a unified embedding space, we perform for the
first time transfer learning between search spaces. Finally, we show the
evolution of embeddings during training, motivating future studies into using
embeddings at different training stages to gain a deeper understanding of the
networks in a search space.
Related papers
- EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition [54.99121380536659]
Eye movement biometrics have received increasing attention thanks to its high secure identification.
Deep learning (DL) models have been recently successfully applied for eye movement recognition.
DL architecture still is determined by human prior knowledge.
We propose EM-DARTS, a hierarchical differentiable architecture search algorithm to automatically design the DL architecture for eye movement recognition.
arXiv Detail & Related papers (2024-09-22T13:11:08Z) - OFA$^2$: A Multi-Objective Perspective for the Once-for-All Neural
Architecture Search [79.36688444492405]
Once-for-All (OFA) is a Neural Architecture Search (NAS) framework designed to address the problem of searching efficient architectures for devices with different resources constraints.
We aim to give one step further in the search for efficiency by explicitly conceiving the search stage as a multi-objective optimization problem.
arXiv Detail & Related papers (2023-03-23T21:30:29Z) - Searching a High-Performance Feature Extractor for Text Recognition
Network [92.12492627169108]
We design a domain-specific search space by exploring principles for having good feature extractors.
As the space is huge and complexly structured, no existing NAS algorithms can be applied.
We propose a two-stage algorithm to effectively search in the space.
arXiv Detail & Related papers (2022-09-27T03:49:04Z) - Search Space Adaptation for Differentiable Neural Architecture Search in
Image Classification [15.641353388251465]
Differentiable neural architecture search (NAS) has a great impact by reducing the search cost to the level of training a single network.
In this paper, we propose an adaptation scheme of the search space by introducing a search scope.
The effectiveness of proposed method is demonstrated with ProxylessNAS for the image classification task.
arXiv Detail & Related papers (2022-06-05T05:27:12Z) - Neural Architecture Search for Speech Emotion Recognition [72.1966266171951]
We propose to apply neural architecture search (NAS) techniques to automatically configure the SER models.
We show that NAS can improve SER performance (54.89% to 56.28%) while maintaining model parameter sizes.
arXiv Detail & Related papers (2022-03-31T10:16:10Z) - NeuralArTS: Structuring Neural Architecture Search with Type Theory [0.0]
We present a new framework called Neural Architecture Type System (NeuralArTS) that categorizes the infinite set of network operations in a structured type system.
We show how NeuralArTS can be applied to convolutional layers and propose several future directions.
arXiv Detail & Related papers (2021-10-17T03:28:27Z) - Task-Aware Neural Architecture Search [33.11791812491669]
We propose a novel framework for neural architecture search, utilizing a dictionary of models of base tasks and the similarity between the target task and the atoms of the dictionary.
By introducing a gradient-based search algorithm, we can evaluate and discover the best architecture in the search space without fully training the networks.
arXiv Detail & Related papers (2020-10-27T00:10:40Z) - NAS-Navigator: Visual Steering for Explainable One-Shot Deep Neural
Network Synthesis [53.106414896248246]
We present a framework that allows analysts to effectively build the solution sub-graph space and guide the network search by injecting their domain knowledge.
Applying this technique in an iterative manner allows analysts to converge to the best performing neural network architecture for a given application.
arXiv Detail & Related papers (2020-09-28T01:48:45Z) - NAS-DIP: Learning Deep Image Prior with Neural Architecture Search [65.79109790446257]
Recent work has shown that the structure of deep convolutional neural networks can be used as a structured image prior.
We propose to search for neural architectures that capture stronger image priors.
We search for an improved network by leveraging an existing neural architecture search algorithm.
arXiv Detail & Related papers (2020-08-26T17:59:36Z) - An Introduction to Neural Architecture Search for Convolutional Networks [0.0]
Neural Architecture Search (NAS) is a research field concerned with utilizing optimization algorithms to design optimal neural network architectures.
We provide an introduction to the basic concepts of NAS for convolutional networks, along with the major advances in search spaces, algorithms and evaluation techniques.
arXiv Detail & Related papers (2020-05-22T09:33:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.