Transferring Core Knowledge via Learngenes
- URL: http://arxiv.org/abs/2401.08139v1
- Date: Tue, 16 Jan 2024 06:18:11 GMT
- Title: Transferring Core Knowledge via Learngenes
- Authors: Fu Feng, Jing Wang and Xin Geng
- Abstract summary: We propose the Genetic Transfer Learning (GTL) framework to copy the evolutionary process of organisms into neural networks.
GTL trains a population of networks, selects superior learngenes by tournaments, performs learngene mutations, and passes the learngenes to next generations.
We show that the learngenes bring the descendant networks instincts and strong learning ability.
- Score: 45.651726289932334
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The pre-training paradigm fine-tunes the models trained on large-scale
datasets to downstream tasks with enhanced performance. It transfers all
knowledge to downstream tasks without discriminating which part is necessary or
unnecessary, which may lead to negative transfer. In comparison, knowledge
transfer in nature is much more efficient. When passing genetic information to
descendants, ancestors encode only the essential knowledge into genes, which
act as the medium. Inspired by that, we adopt a recent concept called
``learngene'' and refine its structures by mimicking the structures of natural
genes. We propose the Genetic Transfer Learning (GTL) -- a framework to copy
the evolutionary process of organisms into neural networks. GTL trains a
population of networks, selects superior learngenes by tournaments, performs
learngene mutations, and passes the learngenes to next generations. Finally, we
successfully extract the learngenes of VGG11 and ResNet12. We show that the
learngenes bring the descendant networks instincts and strong learning ability:
with 20% parameters, the learngenes bring 12% and 16% improvements of accuracy
on CIFAR-FS and miniImageNet. Besides, the learngenes have the scalability and
adaptability on the downstream structure of networks and datasets. Overall, we
offer a novel insight that transferring core knowledge via learngenes may be
sufficient and efficient for neural networks.
Related papers
- Memory Networks: Towards Fully Biologically Plausible Learning [2.7013801448234367]
Current artificial neural networks rely on techniques like backpropagation and weight sharing, which do not align with the brain's natural information processing methods.
We propose the Memory Network, a model inspired by biological principles that avoids backpropagation and convolutions, and operates in a single pass.
arXiv Detail & Related papers (2024-09-18T06:01:35Z) - Breaching the Bottleneck: Evolutionary Transition from Reward-Driven Learning to Reward-Agnostic Domain-Adapted Learning in Neuromodulated Neural Nets [0.3428444467046466]
AI learning algorithms rely on explicit externally provided measures of behaviour quality to acquire fit behaviour.
This imposes an information bottleneck that precludes learning from diverse non-reward stimulus information.
We propose that species first evolve the ability to learn from reward signals, providing inefficient (bottlenecked) but broad adaptivity.
arXiv Detail & Related papers (2024-04-19T05:14:47Z) - Genetic Programming-Based Evolutionary Deep Learning for Data-Efficient
Image Classification [3.9310727060473476]
This paper proposes a new genetic programming-based evolutionary deep learning approach to data-efficient image classification.
The new approach can automatically evolve variable-length models using many important operators from both image and classification domains.
A flexible multi-layer representation enables the new approach to automatically construct shallow or deep models/trees for different tasks.
arXiv Detail & Related papers (2022-09-27T08:10:16Z) - A Neural Network Based Method with Transfer Learning for Genetic Data
Analysis [3.8599966694228667]
We combine transfer learning technique with a neural network based method(expectile neural networks)
We leverage previous learnings and avoid starting from scratch to improve the model performance.
By using transfer learning algorithm, the performance of expectile neural networks is improved compared to expectile neural network without using transfer learning technique.
arXiv Detail & Related papers (2022-06-20T16:16:05Z) - aSTDP: A More Biologically Plausible Learning [0.0]
We introduce approximate STDP, a new neural networks learning framework.
It uses only STDP rules for supervised and unsupervised learning.
It can make predictions or generate patterns in one model without additional configuration.
arXiv Detail & Related papers (2022-05-22T08:12:50Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - A Few-shot Learning Graph Multi-Trajectory Evolution Network for
Forecasting Multimodal Baby Connectivity Development from a Baseline
Timepoint [53.73316520733503]
We propose a Graph Multi-Trajectory Evolution Network (GmTE-Net), which adopts a teacher-student paradigm.
This is the first teacher-student architecture tailored for brain graph multi-trajectory growth prediction.
arXiv Detail & Related papers (2021-10-06T08:26:57Z) - Adversarially-Trained Deep Nets Transfer Better: Illustration on Image
Classification [53.735029033681435]
Transfer learning is a powerful methodology for adapting pre-trained deep neural networks on image recognition tasks to new domains.
In this work, we demonstrate that adversarially-trained models transfer better than non-adversarially-trained models.
arXiv Detail & Related papers (2020-07-11T22:48:42Z) - Curriculum By Smoothing [52.08553521577014]
Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation.
We propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters.
As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data.
arXiv Detail & Related papers (2020-03-03T07:27:44Z) - Alpha Discovery Neural Network based on Prior Knowledge [55.65102700986668]
Genetic programming (GP) is the state-of-the-art in financial automated feature construction task.
This paper proposes Alpha Discovery Neural Network (ADNN), a tailored neural network structure which can automatically construct diversified financial technical indicators.
arXiv Detail & Related papers (2019-12-26T03:10:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.