Neural Architecture Search for Lightweight Non-Local Networks

• URL: http://arxiv.org/abs/2004.01961v1
• Date: Sat, 4 Apr 2020 15:46:39 GMT
• Title: Neural Architecture Search for Lightweight Non-Local Networks
• Authors: Yingwei Li, Xiaojie Jin, Jieru Mei, Xiaochen Lian, Linjie Yang, Cihang Xie, Qihang Yu, Yuyin Zhou, Song Bai, Alan Yuille
• Abstract summary: Non-Local (NL) blocks have been widely studied in various vision tasks. We propose a Lightweight Non-Local (LightNL) block by squeezing the transformation operations and incorporating compact features. We also propose an efficient neural architecture search algorithm to learn an optimal configuration of LightNL blocks in an end-to-end manner.
• Score: 66.49621237326959
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Non-Local (NL) blocks have been widely studied in various vision tasks. However, it has been rarely explored to embed the NL blocks in mobile neural networks, mainly due to the following challenges: 1) NL blocks generally have heavy computation cost which makes it difficult to be applied in applications where computational resources are limited, and 2) it is an open problem to discover an optimal configuration to embed NL blocks into mobile neural networks. We propose AutoNL to overcome the above two obstacles. Firstly, we propose a Lightweight Non-Local (LightNL) block by squeezing the transformation operations and incorporating compact features. With the novel design choices, the proposed LightNL block is 400x computationally cheaper} than its conventional counterpart without sacrificing the performance. Secondly, by relaxing the structure of the LightNL block to be differentiable during training, we propose an efficient neural architecture search algorithm to learn an optimal configuration of LightNL blocks in an end-to-end manner. Notably, using only 32 GPU hours, the searched AutoNL model achieves 77.7% top-1 accuracy on ImageNet under a typical mobile setting (350M FLOPs), significantly outperforming previous mobile models including MobileNetV2 (+5.7%), FBNet (+2.8%) and MnasNet (+2.1%). Code and models are available at https://github.com/LiYingwei/AutoNL.

Related papers

• Energy-Aware FPGA Implementation of Spiking Neural Network with LIF Neurons [0.5243460995467893]
  Spiking Neural Networks (SNNs) stand out as a cutting-edge solution for TinyML. This paper presents a novel SNN architecture based on the 1st Order Leaky Integrate-and-Fire (LIF) neuron model. A hardware-friendly LIF design is also proposed, and implemented on a Xilinx Artix-7 FPGA.
  arXiv  Detail & Related papers  (2024-11-03T16:42:10Z)
• Unlocking FedNL: Self-Contained Compute-Optimized Implementation [56.16884466478886]
  Federated Learning (FL) is an emerging paradigm that enables intelligent agents to collaboratively train Machine Learning (ML) models in a distributed manner. Recent work introduces a family of Federated Newton Learn (FedNL) algorithms, marking a significant step towards applying second-order methods to FL and large-scale optimization. We present a self-contained implementation of FedNL, FedNL-LS, FedNL-PP for single-node and multi-node settings.
  arXiv  Detail & Related papers  (2024-10-11T12:19:18Z)
• A foundation for exact binarized morphological neural networks [2.8925699537310137]
  Training and running deep neural networks (NNs) often demands a lot of computation and energy-intensive specialized hardware. One way to reduce the computation and power cost is to use binary weight NNs, but these are hard to train because the sign function has a non-smooth gradient. We present a model based on Mathematical Morphology (MM), which can binarize ConvNets without losing performance under certain conditions.
  arXiv  Detail & Related papers  (2024-01-08T11:37:44Z)
• Latency-aware Unified Dynamic Networks for Efficient Image Recognition [72.8951331472913]
  LAUDNet is a framework to bridge the theoretical and practical efficiency gap in dynamic networks. It integrates three primary dynamic paradigms-spatially adaptive computation, dynamic layer skipping, and dynamic channel skipping. It can notably reduce the latency of models like ResNet by over 50% on platforms such as V100,3090, and TX2 GPUs.
  arXiv  Detail & Related papers  (2023-08-30T10:57:41Z)
• Rethinking Efficacy of Softmax for Lightweight Non-Local Neural Networks [22.240253892754932]
  Non-local (NL) block is a popular module that demonstrates the capability to model global contexts. We empirically analyze if the magnitude and direction of input feature vectors properly affect the attention between vectors. By replacing softmax operation with the scaling factor, we demonstrate improved performance on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
  arXiv  Detail & Related papers  (2022-07-27T10:04:23Z)
• Neural network relief: a pruning algorithm based on neural activity [47.57448823030151]
  We propose a simple importance-score metric that deactivates unimportant connections. We achieve comparable performance for LeNet architectures on MNIST. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations.
  arXiv  Detail & Related papers  (2021-09-22T15:33:49Z)
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration [83.84684675841167]
  We propose a novel encoding scheme using -1, +1 to decompose quantized neural networks (QNNs) into multi-branch binary networks. We validate the effectiveness of our method on large-scale image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-06-18T03:11:15Z)
• FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation [81.76975488010213]
  Dense optical flow estimation plays a key role in many robotic vision tasks. Current networks often occupy large number of parameters and require heavy computation costs. Our proposed FastFlowNet works in the well-known coarse-to-fine manner with following innovations.
  arXiv  Detail & Related papers  (2021-03-08T03:09:37Z)
• LRNNet: A Light-Weighted Network with Efficient Reduced Non-Local Operation for Real-Time Semantic Segmentation [15.010572800399057]
  This paper introduces a light-weighted network with an efficient reduced non-local module (LRNNet) for efficient and realtime semantic segmentation. Experiments demonstrate our superior trade-off among light-weight, speed, computation and accuracy.
  arXiv  Detail & Related papers  (2020-06-04T08:55:15Z)
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy? [12.050205584630922]
  Binary Neural Networks (BNNs) are neural networks which use binary weights and activations instead of the typical 32-bit floating point values. In this paper, we present an architectural approach: MeliusNet. It consists of alternating a DenseBlock, which increases the feature capacity, and our proposed ImprovementBlock, which increases the feature quality.
  arXiv  Detail & Related papers  (2020-01-16T16:56:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.