Growing Neural Network with Shared Parameter

• URL: http://arxiv.org/abs/2201.06500v1
• Date: Mon, 17 Jan 2022 16:24:17 GMT
• Title: Growing Neural Network with Shared Parameter
• Authors: Ruilin Tong
• Abstract summary: We propose a general method for growing neural network with shared parameter by matching trained network to new input. Our method has shown the ability to improve performance with higher parameter efficiency. It can also be applied to trans-task case and realize transfer learning by changing the combination ofworks without training on new task.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a general method for growing neural network with shared parameter by matching trained network to new input. By leveraging Hoeffding's inequality, we provide a theoretical base for improving performance by adding subnetwork to existing network. With the theoretical base of adding new subnetwork, we implement a matching method to apply trained subnetwork of existing network to new input. Our method has shown the ability to improve performance with higher parameter efficiency. It can also be applied to trans-task case and realize transfer learning by changing the combination of subnetworks without training on new task.

Related papers

• Stitching for Neuroevolution: Recombining Deep Neural Networks without Breaking Them [0.0]
  Traditional approaches to neuroevolution often start from scratch. Recombining trained networks is non-trivial because architectures and feature representations typically differ. We employ stitching, which merges the networks by introducing new layers at crossover points.
  arXiv  Detail & Related papers  (2024-03-21T08:30:44Z)
• Adaptive Depth Networks with Skippable Sub-Paths [1.8416014644193066]
  We present a practical approach to adaptive depth networks with minimal training effort. Our approach does not train every target sub-network in an iterative manner. We provide a formal rationale for why the proposed training method can reduce overall prediction errors.
  arXiv  Detail & Related papers  (2023-12-27T03:43:38Z)
• Stimulative Training++: Go Beyond The Performance Limits of Residual Networks [91.5381301894899]
  Residual networks have shown great success and become indispensable in recent deep neural network models. Previous research has suggested that residual networks can be considered as ensembles of shallow networks. We identify a problem that is analogous to social loafing, whereworks within a residual network are prone to exert less effort when working as part of a group compared to working alone.
  arXiv  Detail & Related papers  (2023-05-04T02:38:11Z)
• Increasing Depth of Neural Networks for Life-long Learning [2.0305676256390934]
  We propose a novel method for continual learning based on the increasing depth of neural networks. This work explores whether extending neural network depth may be beneficial in a life-long learning setting.
  arXiv  Detail & Related papers  (2022-02-22T11:21:41Z)
• Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization [8.661269034961679]
  We propose a biologically inspired method for improving the computational resource utilization of neural networks. The proposed method utilizes the channel activations of a convolution layer in order to reorganize that layers parameters. The rejuvenated parameters learn different features to supplement those learned by the reorganized surviving parameters.
  arXiv  Detail & Related papers  (2021-02-13T06:19:45Z)
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks [94.69202357137452]
  We propose a novel model-parallel learning method, called local critic training. We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) We also show that trained networks by the proposed method can be used for structural optimization.
  arXiv  Detail & Related papers  (2021-02-03T09:30:45Z)
• Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks [78.47459801017959]
  Sparsity can reduce the memory footprint of regular networks to fit mobile devices. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice.
  arXiv  Detail & Related papers  (2021-01-31T22:48:50Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)
• Semantic Drift Compensation for Class-Incremental Learning [48.749630494026086]
  Class-incremental learning of deep networks sequentially increases the number of classes to be classified. We propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars.
  arXiv  Detail & Related papers  (2020-04-01T13:31:19Z)
• Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks [95.51368472949308]
  Adaptation can be useful in cases when training data is scarce, or when one wishes to encode priors in the network. In this paper, we propose a straightforward alternative: side-tuning.
  arXiv  Detail & Related papers  (2019-12-31T18:52:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.