Neural Routing in Meta Learning
- URL: http://arxiv.org/abs/2210.07932v1
- Date: Fri, 14 Oct 2022 16:31:24 GMT
- Title: Neural Routing in Meta Learning
- Authors: Jicang Cai, Saeed Vahidian, Weijia Wang, Mohsen Joneidi, and Bill Lin
- Abstract summary: We aim to improve the model performance of the current meta learning algorithms by selectively using only parts of the model conditioned on the input tasks.
In this work, we describe an approach that investigates task-dependent dynamic neuron selection in deep convolutional neural networks (CNNs) by leveraging the scaling factor in the batch normalization layer.
We find that the proposed approach, neural routing in meta learning (NRML), outperforms one of the well-known existing meta learning baselines on few-shot classification tasks.
- Score: 9.070747377130472
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Meta-learning often referred to as learning-to-learn is a promising notion
raised to mimic human learning by exploiting the knowledge of prior tasks but
being able to adapt quickly to novel tasks. A plethora of models has emerged in
this context and improved the learning efficiency, robustness, etc. The
question that arises here is can we emulate other aspects of human learning and
incorporate them into the existing meta learning algorithms? Inspired by the
widely recognized finding in neuroscience that distinct parts of the brain are
highly specialized for different types of tasks, we aim to improve the model
performance of the current meta learning algorithms by selectively using only
parts of the model conditioned on the input tasks. In this work, we describe an
approach that investigates task-dependent dynamic neuron selection in deep
convolutional neural networks (CNNs) by leveraging the scaling factor in the
batch normalization (BN) layer associated with each convolutional layer. The
problem is intriguing because the idea of helping different parts of the model
to learn from different types of tasks may help us train better filters in
CNNs, and improve the model generalization performance. We find that the
proposed approach, neural routing in meta learning (NRML), outperforms one of
the well-known existing meta learning baselines on few-shot classification
tasks on the most widely used benchmark datasets.
Related papers
- Similarity-based context aware continual learning for spiking neural networks [12.259720271932661]
We propose a Similarity-based Context Aware Spiking Neural Network (SCA-SNN) continual learning algorithm.
Based on contextual similarity across tasks, the SCA-SNN model can adaptively reuse neurons from previous tasks that are beneficial for new tasks.
Our algorithm has the capability to adaptively select similar groups of neurons for related tasks, offering a promising approach to enhancing the biological interpretability of efficient continual learning.
arXiv Detail & Related papers (2024-10-28T09:38:57Z) - Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
This paper introduces the SANE approach to weight-space learning.
Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
arXiv Detail & Related papers (2024-06-14T13:12:07Z) - Image classification network enhancement methods based on knowledge
injection [8.885876832491917]
This paper proposes a multi-level hierarchical deep learning algorithm.
It is composed of multi-level hierarchical deep neural network architecture and multi-level hierarchical deep learning framework.
The experimental results show that the proposed algorithm can effectively explain the hidden information of the neural network.
arXiv Detail & Related papers (2024-01-09T09:11:41Z) - Sparse Multitask Learning for Efficient Neural Representation of Motor
Imagery and Execution [30.186917337606477]
We introduce a sparse multitask learning framework for motor imagery (MI) and motor execution (ME) tasks.
Given a dual-task CNN model for MI-ME classification, we apply a saliency-based sparsification approach to prune superfluous connections.
Our results indicate that this tailored sparsity can mitigate the overfitting problem and improve the test performance with small amount of data.
arXiv Detail & Related papers (2023-12-10T09:06:16Z) - Meta-Learning in Spiking Neural Networks with Reward-Modulated STDP [2.179313476241343]
We propose a bio-plausible meta-learning model inspired by the hippocampus and the prefrontal cortex.
Our new model can easily be applied to spike-based neuromorphic devices and enables fast learning in neuromorphic hardware.
arXiv Detail & Related papers (2023-06-07T13:08:46Z) - Can we learn gradients by Hamiltonian Neural Networks? [68.8204255655161]
We propose a meta-learner based on ODE neural networks that learns gradients.
We demonstrate that our method outperforms a meta-learner based on LSTM for an artificial task and the MNIST dataset with ReLU activations in the optimizee.
arXiv Detail & Related papers (2021-10-31T18:35:10Z) - Dynamic Neural Diversification: Path to Computationally Sustainable
Neural Networks [68.8204255655161]
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks.
We explore the diversity of the neurons within the hidden layer during the learning process.
We analyze how the diversity of the neurons affects predictions of the model.
arXiv Detail & Related papers (2021-09-20T15:12:16Z) - Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
There is an increasing need for active learning algorithms that are compatible with deep neural networks.
This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
arXiv Detail & Related papers (2021-06-17T17:26:31Z) - Incremental Training of a Recurrent Neural Network Exploiting a
Multi-Scale Dynamic Memory [79.42778415729475]
We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning.
We show how to extend the architecture of a simple RNN by separating its hidden state into different modules.
We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
arXiv Detail & Related papers (2020-06-29T08:35:49Z) - Towards Efficient Processing and Learning with Spikes: New Approaches
for Multi-Spike Learning [59.249322621035056]
We propose two new multi-spike learning rules which demonstrate better performance over other baselines on various tasks.
In the feature detection task, we re-examine the ability of unsupervised STDP with its limitations being presented.
Our proposed learning rules can reliably solve the task over a wide range of conditions without specific constraints being applied.
arXiv Detail & Related papers (2020-05-02T06:41:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.