DartsReNet: Exploring new RNN cells in ReNet architectures

• URL: http://arxiv.org/abs/2304.05838v1
• Date: Tue, 11 Apr 2023 09:42:10 GMT
• Title: DartsReNet: Exploring new RNN cells in ReNet architectures
• Authors: Brian Moser, Federico Raue, J\"orn Hees, Andreas Dengel
• Abstract summary: We present new Recurrent Neural Network (RNN) cells for image classification using a Neural Architecture Search (NAS) approach called DARTS. We are interested in the ReNet architecture, which is a RNN based approach presented as an alternative for convolutional and pooling steps.
• Score: 4.266320191208303
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present new Recurrent Neural Network (RNN) cells for image classification using a Neural Architecture Search (NAS) approach called DARTS. We are interested in the ReNet architecture, which is a RNN based approach presented as an alternative for convolutional and pooling steps. ReNet can be defined using any standard RNN cells, such as LSTM and GRU. One limitation is that standard RNN cells were designed for one dimensional sequential data and not for two dimensions like it is the case for image classification. We overcome this limitation by using DARTS to find new cell designs. We compare our results with ReNet that uses GRU and LSTM cells. Our found cells outperform the standard RNN cells on CIFAR-10 and SVHN. The improvements on SVHN indicate generalizability, as we derived the RNN cell designs from CIFAR-10 without performing a new cell search for SVHN.

Related papers

• Were RNNs All We Needed? [53.393497486332]
  We revisit traditional recurrent neural networks (RNNs) from over a decade ago. We show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel.
  arXiv  Detail & Related papers  (2024-10-02T03:06:49Z)
• Recurrent Neural Networks for Still Images [0.0]
  We argue that RNNs can effectively handle still images by interpreting the pixels as a sequence. We introduce a novel RNN design tailored for two-dimensional inputs, such as images, and a custom version of BiDirectional RNN (BiRNN) that is more memory-efficient than traditional implementations.
  arXiv  Detail & Related papers  (2024-09-10T06:07:20Z)
• scBiGNN: Bilevel Graph Representation Learning for Cell Type Classification from Single-cell RNA Sequencing Data [62.87454293046843]
  Graph neural networks (GNNs) have been widely used for automatic cell type classification. scBiGNN comprises two GNN modules to identify cell types. scBiGNN outperforms a variety of existing methods for cell type classification from scRNA-seq data.
  arXiv  Detail & Related papers  (2023-12-16T03:54:26Z)
• Decomposing a Recurrent Neural Network into Modules for Enabling Reusability and Replacement [11.591247347259317]
  We propose the first approach to decompose an RNN into modules. We study different types of RNNs, i.e., Vanilla, LSTM, and GRU. We show how such RNN modules can be reused and replaced in various scenarios.
  arXiv  Detail & Related papers  (2022-12-09T03:29:38Z)
• Sequence Transduction with Graph-based Supervision [96.04967815520193]
  We present a new transducer objective function that generalizes the RNN-T loss to accept a graph representation of the labels. We demonstrate that transducer-based ASR with CTC-like lattice achieves better results compared to standard RNN-T.
  arXiv  Detail & Related papers  (2021-11-01T21:51:42Z)
• Gates Are Not What You Need in RNNs [2.6199029802346754]
  We propose a new recurrent cell called Residual Recurrent Unit (RRU) which beats traditional cells and does not employ a single gate. It is based on the residual shortcut connection, linear transformations, ReLU, and normalization. Our experiments show that RRU outperforms the traditional gated units on most of these tasks.
  arXiv  Detail & Related papers  (2021-08-01T19:20:34Z)
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN [9.20540910698296]
  We discuss the similarities between recurrent neural network (RNN) and serial adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module to RNN, which makes it possible for RNN to run in parallel.
  arXiv  Detail & Related papers  (2021-06-22T12:28:33Z)
• Convolutional Neural Networks with Gated Recurrent Connections [25.806036745901114]
  recurrent convolution neural network (RCNN) is inspired by abundant recurrent connections in the visual systems of animals. We propose to modulate the receptive fields (RFs) of neurons by introducing gates to the recurrent connections. The GRCNN was evaluated on several computer vision tasks including object recognition, scene text recognition and object detection.
  arXiv  Detail & Related papers  (2021-06-05T10:14:59Z)
• Stretchable Cells Help DARTS Search Better [70.52254306274092]
  Differentiable neural architecture search (DARTS) has gained much success in discovering flexible and diverse cell types. Current DARTS methods are prone to wide and shallow cells, and this topology collapse induces sub-optimal searched cells. In this paper, we endowing the cells with explicit stretchability, so the search can be directly implemented on our stretchable cells.
  arXiv  Detail & Related papers  (2020-11-18T14:15:51Z)
• Fusion Recurrent Neural Network [88.5550074808201]
  We propose a novel, succinct and promising RNN - Fusion Recurrent Neural Network (Fusion RNN) Fusion RNN is composed of Fusion module and Transport module every time step. In order to evaluate Fusion RNN's sequence feature extraction capability, we choose a representative data mining task for sequence data, estimated time of arrival (ETA) and present a novel model based on Fusion RNN.
  arXiv  Detail & Related papers  (2020-06-07T07:39:49Z)
• Visual Commonsense R-CNN [102.5061122013483]
  We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN) VC R-CNN serves as an improved visual region encoder for high-level tasks such as captioning and VQA. We extensively apply VC R-CNN features in prevailing models of three popular tasks: Image Captioning, VQA, and VCR, and observe consistent performance boosts across them.
  arXiv  Detail & Related papers  (2020-02-27T15:51:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.