A Novel Approach to Lifelong Learning: The Plastic Support Structure
- URL: http://arxiv.org/abs/2106.06298v1
- Date: Fri, 11 Jun 2021 10:34:37 GMT
- Title: A Novel Approach to Lifelong Learning: The Plastic Support Structure
- Authors: Georges Kanaan, Kai Wen Zheng and Lucas Fenaux
- Abstract summary: We propose a novel approach to lifelong learning, introducing a compact encapsulated support structure which endows a network with the capability to expand its capacity as needed to learn new tasks.
This is achieved by splitting neurons with high semantic drift and constructing an adjacent network to encode the new tasks at hand.
We call this the Plastic Support Structure (PSS), it is a compact structure to learn new tasks that cannot be efficiently encoded in the existing structure of the network.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a novel approach to lifelong learning, introducing a compact
encapsulated support structure which endows a network with the capability to
expand its capacity as needed to learn new tasks while preventing the loss of
learned tasks. This is achieved by splitting neurons with high semantic drift
and constructing an adjacent network to encode the new tasks at hand. We call
this the Plastic Support Structure (PSS), it is a compact structure to learn
new tasks that cannot be efficiently encoded in the existing structure of the
network. We validate the PSS on public datasets against existing lifelong
learning architectures, showing it performs similarly to them but without prior
knowledge of the task and in some cases with fewer parameters and in a more
understandable fashion where the PSS is an encapsulated container for specific
features related to specific tasks, thus making it an ideal "add-on" solution
for endowing a network to learn more tasks.
Related papers
- Aux-NAS: Exploiting Auxiliary Labels with Negligibly Extra Inference Cost [73.28626942658022]
We aim at exploiting additional auxiliary labels from an independent (auxiliary) task to boost the primary task performance.
Our method is architecture-based with a flexible asymmetric structure for the primary and auxiliary tasks.
Experiments with six tasks on NYU v2, CityScapes, and Taskonomy datasets using VGG, ResNet, and ViT backbones validate the promising performance.
arXiv Detail & Related papers (2024-05-09T11:50:19Z) - Dense Network Expansion for Class Incremental Learning [61.00081795200547]
State-of-the-art approaches use a dynamic architecture based on network expansion (NE), in which a task expert is added per task.
A new NE method, dense network expansion (DNE), is proposed to achieve a better trade-off between accuracy and model complexity.
It outperforms the previous SOTA methods by a margin of 4% in terms of accuracy, with similar or even smaller model scale.
arXiv Detail & Related papers (2023-03-22T16:42:26Z) - PDSketch: Integrated Planning Domain Programming and Learning [86.07442931141637]
We present a new domain definition language, named PDSketch.
It allows users to flexibly define high-level structures in the transition models.
Details of the transition model will be filled in by trainable neural networks.
arXiv Detail & Related papers (2023-03-09T18:54:12Z) - Break It Down: Evidence for Structural Compositionality in Neural
Networks [32.382094867951224]
We show that neural networks can learn compositionality, obviating the need for specialized symbolic mechanisms.
This suggests that neural networks may be able to learn compositionality, obviating the need for specialized symbolic mechanisms.
arXiv Detail & Related papers (2023-01-26T00:53:11Z) - E2-AEN: End-to-End Incremental Learning with Adaptively Expandable
Network [57.87240860624937]
We propose an end-to-end trainable adaptively expandable network named E2-AEN.
It dynamically generates lightweight structures for new tasks without any accuracy drop in previous tasks.
E2-AEN reduces cost and can be built upon any feed-forward architectures in an end-to-end manner.
arXiv Detail & Related papers (2022-07-14T09:04:51Z) - Fast Inference and Transfer of Compositional Task Structures for
Few-shot Task Generalization [101.72755769194677]
We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph.
Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks.
Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
arXiv Detail & Related papers (2022-05-25T10:44:25Z) - Increasing Depth of Neural Networks for Life-long Learning [2.0305676256390934]
We propose a novel method for continual learning based on the increasing depth of neural networks.
This work explores whether extending neural network depth may be beneficial in a life-long learning setting.
arXiv Detail & Related papers (2022-02-22T11:21:41Z) - Learn to Bind and Grow Neural Structures [0.3553493344868413]
We present a new framework, Learn to Bind and Grow, which learns a neural architecture for a new task incrementally.
Central to our approach is a novel, interpretable, parameterization of the shared, multi-task architecture space.
Experiments on continual learning benchmarks show that our framework performs comparably with earlier expansion based approaches.
arXiv Detail & Related papers (2020-11-21T09:40:26Z) - Adversarial Continual Learning [99.56738010842301]
We propose a hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features.
Our model combines architecture growth to prevent forgetting of task-specific skills and an experience replay approach to preserve shared skills.
arXiv Detail & Related papers (2020-03-21T02:08:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.