A Study of Continual Learning Methods for Q-Learning
- URL: http://arxiv.org/abs/2206.03934v1
- Date: Wed, 8 Jun 2022 14:51:52 GMT
- Title: A Study of Continual Learning Methods for Q-Learning
- Authors: Benedikt Bagus and Alexander Gepperth
- Abstract summary: We present an empirical study on the use of continual learning (CL) methods in a reinforcement learning (RL) scenario.
Our results show that dedicated CL methods can significantly improve learning when compared to the baseline technique of "experience replay"
- Score: 78.6363825307044
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present an empirical study on the use of continual learning (CL) methods
in a reinforcement learning (RL) scenario, which, to the best of our knowledge,
has not been described before. CL is a very active recent research topic
concerned with machine learning under non-stationary data distributions.
Although this naturally applies to RL, the use of dedicated CL methods is still
uncommon. This may be due to the fact that CL methods often assume a
decomposition of CL problems into disjoint sub-tasks of stationary
distribution, that the onset of these sub-tasks is known, and that sub-tasks
are non-contradictory. In this study, we perform an empirical comparison of
selected CL methods in a RL problem where a physically simulated robot must
follow a racetrack by vision. In order to make CL methods applicable, we
restrict the RL setting and introduce non-conflicting subtasks of known onset,
which are however not disjoint and whose distribution, from the learner's point
of view, is still non-stationary. Our results show that dedicated CL methods
can significantly improve learning when compared to the baseline technique of
"experience replay".
Related papers
- Continual Task Learning through Adaptive Policy Self-Composition [54.95680427960524]
CompoFormer is a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network.
Our experiments reveal that CompoFormer outperforms conventional continual learning (CL) methods, particularly in longer task sequences.
arXiv Detail & Related papers (2024-11-18T08:20:21Z) - ICL-TSVD: Bridging Theory and Practice in Continual Learning with Pre-trained Models [103.45785408116146]
Continual learning (CL) aims to train a model that can solve multiple tasks presented sequentially.
Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks.
However, such methods lack theoretical guarantees, making them prone to unexpected failures.
We bridge this gap by integrating an empirically strong approach into a principled framework, designed to prevent forgetting.
arXiv Detail & Related papers (2024-10-01T12:58:37Z) - Can Continual Learning Improve Long-Tailed Recognition? Toward a Unified
Framework [16.457778420360537]
Long-Tailed Recognition methods aim to accurately learn a dataset comprising both a larger Head set and a smaller Tail set.
We show that Continual Learning (CL) methods can effectively update the weights of the learner to learn the Tail without forgetting the Head.
We also assess the applicability of CL techniques on real-world data by exploring CL on the naturally imbalanced256 dataset.
arXiv Detail & Related papers (2023-06-23T03:05:33Z) - CCL: Continual Contrastive Learning for LiDAR Place Recognition [5.025654873456756]
Current deep learning-based methods suffer from poor generalization ability and catastrophic forgetting.
We propose a continual contrastive learning method, named CCL, to tackle the catastrophic forgetting problem.
Our method consistently improves the performance of different methods in different environments outperforming the state-of-the-art continual learning method.
arXiv Detail & Related papers (2023-03-24T12:14:54Z) - Real-Time Evaluation in Online Continual Learning: A New Hope [104.53052316526546]
We evaluate current Continual Learning (CL) methods with respect to their computational costs.
A simple baseline outperforms state-of-the-art CL methods under this evaluation.
This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical.
arXiv Detail & Related papers (2023-02-02T12:21:10Z) - Beyond Supervised Continual Learning: a Review [69.9674326582747]
Continual Learning (CL) is a flavor of machine learning where the usual assumption of stationary data distribution is relaxed or omitted.
Changes in the data distribution can cause the so-called catastrophic forgetting (CF) effect: an abrupt loss of previous knowledge.
This article reviews literature that study CL in other settings, such as learning with reduced supervision, fully unsupervised learning, and reinforcement learning.
arXiv Detail & Related papers (2022-08-30T14:44:41Z) - Weakly Supervised Continual Learning [17.90483695137098]
This work explores Weakly Supervised Continual Learning (WSCL)
We show that not only our proposals exhibit higher flexibility when supervised information is scarce, but also that less than 25% labels can be enough to reach or even outperform SOTA methods trained under full supervision.
In doing so, we show that not only our proposals exhibit higher flexibility when supervised information is scarce, but also that less than 25% labels can be enough to reach or even outperform SOTA methods trained under full supervision.
arXiv Detail & Related papers (2021-08-14T14:38:20Z) - Continual Learning in Recurrent Neural Networks [67.05499844830231]
We evaluate the effectiveness of continual learning methods for processing sequential data with recurrent neural networks (RNNs)
We shed light on the particularities that arise when applying weight-importance methods, such as elastic weight consolidation, to RNNs.
We show that the performance of weight-importance methods is not directly affected by the length of the processed sequences, but rather by high working memory requirements.
arXiv Detail & Related papers (2020-06-22T10:05:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.