Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networks

• URL: http://arxiv.org/abs/2403.11173v1
• Date: Sun, 17 Mar 2024 11:19:45 GMT
• Title: Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networks
• Authors: Reinhard Booysen, Anna Sergeevna Bosman,
• Abstract summary: This paper proposes a multi-objective evolutionary algorithm-based RNN architecture search method. The proposed method relies on approximate network morphisms for RNN architecture complexity optimisation during evolution.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Artificial neural network (NN) architecture design is a nontrivial and time-consuming task that often requires a high level of human expertise. Neural architecture search (NAS) serves to automate the design of NN architectures and has proven to be successful in automatically finding NN architectures that outperform those manually designed by human experts. NN architecture performance can be quantified based on multiple objectives, which include model accuracy and some NN architecture complexity objectives, among others. The majority of modern NAS methods that consider multiple objectives for NN architecture performance evaluation are concerned with automated feed forward NN architecture design, which leaves multi-objective automated recurrent neural network (RNN) architecture design unexplored. RNNs are important for modeling sequential datasets, and prominent within the natural language processing domain. It is often the case in real world implementations of machine learning and NNs that a reasonable trade-off is accepted for marginally reduced model accuracy in favour of lower computational resources demanded by the model. This paper proposes a multi-objective evolutionary algorithm-based RNN architecture search method. The proposed method relies on approximate network morphisms for RNN architecture complexity optimisation during evolution. The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.

Related papers

• Scalable Mechanistic Neural Networks [52.28945097811129]
  We propose an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) we reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources.
  arXiv  Detail & Related papers  (2024-10-08T14:27:28Z)
• Training-free Neural Architecture Search for RNNs and Transformers [0.0]
  We develop a new training-free metric, named hidden covariance, that predicts the trained performance of an RNN architecture. We find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search.
  arXiv  Detail & Related papers  (2023-06-01T02:06:13Z)
• A Self-adaptive Neuroevolution Approach to Constructing Deep Neural Network Architectures Across Different Types [5.429458930060452]
  We propose a self-adaptive neuroevolution (SANE) approach to automatically construct various lightweight Deep Neural Network (DNN) architectures for different tasks. One of the key settings in SANE is the search space defined by cells and organs self-adapted to different DNN types. SANE is able to self-adaptively adjust evolution exploration and exploitation to improve search efficiency.
  arXiv  Detail & Related papers  (2022-11-27T07:40:25Z)
• Universal approximation property of invertible neural networks [76.95927093274392]
  Invertible neural networks (INNs) are neural network architectures with invertibility by design. Thanks to their invertibility and the tractability of Jacobian, INNs have various machine learning applications such as probabilistic modeling, generative modeling, and representation learning.
  arXiv  Detail & Related papers  (2022-04-15T10:45:26Z)
• Neural Architecture Search for Spiking Neural Networks [10.303676184878896]
  Spiking Neural Networks (SNNs) have gained huge attention as a potential energy-efficient alternative to conventional Artificial Neural Networks (ANNs) Most prior SNN methods use ANN-like architectures, which could provide sub-optimal performance for temporal sequence processing of binary information in SNNs. We introduce a novel Neural Architecture Search (NAS) approach for finding better SNN architectures.
  arXiv  Detail & Related papers  (2022-01-23T16:34:27Z)
• Composing Recurrent Spiking Neural Networks using Locally-Recurrent Motifs and Risk-Mitigating Architectural Optimization [9.104190653846048]
  In neural circuits, recurrent connectivity plays a crucial role in network function and stability. Existing recurrent spiking neural networks (RSNNs) are often constructed by random connections without optimization. We aim to enable systematic design of large RSNNs via a new scalable RSNN architecture and automated architectural optimization.
  arXiv  Detail & Related papers  (2021-08-04T00:09:39Z)
• Self-Learning for Received Signal Strength Map Reconstruction with Neural Architecture Search [63.39818029362661]
  We present a model based on Neural Architecture Search (NAS) and self-learning for received signal strength ( RSS) map reconstruction. The approach first finds an optimal NN architecture and simultaneously train the deduced model over some ground-truth measurements of a given ( RSS) map. Experimental results show that signal predictions of this second model outperforms non-learning based state-of-the-art techniques and NN models with no architecture search.
  arXiv  Detail & Related papers  (2021-05-17T12:19:22Z)
• Differentiable Neural Architecture Learning for Efficient Neural Network Design [31.23038136038325]
  We introduce a novel emph architecture parameterisation based on scaled sigmoid function. We then propose a general emphiable Neural Architecture Learning (DNAL) method to optimize the neural architecture without the need to evaluate candidate neural networks.
  arXiv  Detail & Related papers  (2021-03-03T02:03:08Z)
• Neural Architecture Search For LF-MMI Trained Time Delay Neural Networks [61.76338096980383]
  A range of neural architecture search (NAS) techniques are used to automatically learn two types of hyper- parameters of state-of-the-art factored time delay neural networks (TDNNs) These include the DARTS method integrating architecture selection with lattice-free MMI (LF-MMI) TDNN training. Experiments conducted on a 300-hour Switchboard corpus suggest the auto-configured systems consistently outperform the baseline LF-MMI TDNN systems.
  arXiv  Detail & Related papers  (2020-07-17T08:32:11Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.