Computationally Budgeted Continual Learning: What Does Matter?
- URL: http://arxiv.org/abs/2303.11165v2
- Date: Sat, 15 Jul 2023 01:49:47 GMT
- Title: Computationally Budgeted Continual Learning: What Does Matter?
- Authors: Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet Dokania, Philip H.S.
Torr, Ser-Nam Lim, Bernard Ghanem, Adel Bibi
- Abstract summary: Continual Learning (CL) aims to sequentially train models on streams of incoming data that vary in distribution by preserving previous knowledge while adapting to new data.
Current CL literature focuses on restricted access to previously seen data, while imposing no constraints on the computational budget for training.
We revisit this problem with a large-scale benchmark and analyze the performance of traditional CL approaches in a compute-constrained setting.
- Score: 128.0827987414154
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Continual Learning (CL) aims to sequentially train models on streams of
incoming data that vary in distribution by preserving previous knowledge while
adapting to new data. Current CL literature focuses on restricted access to
previously seen data, while imposing no constraints on the computational budget
for training. This is unreasonable for applications in-the-wild, where systems
are primarily constrained by computational and time budgets, not storage. We
revisit this problem with a large-scale benchmark and analyze the performance
of traditional CL approaches in a compute-constrained setting, where effective
memory samples used in training can be implicitly restricted as a consequence
of limited computation. We conduct experiments evaluating various CL sampling
strategies, distillation losses, and partial fine-tuning on two large-scale
datasets, namely ImageNet2K and Continual Google Landmarks V2 in data
incremental, class incremental, and time incremental settings. Through
extensive experiments amounting to a total of over 1500 GPU-hours, we find
that, under compute-constrained setting, traditional CL approaches, with no
exception, fail to outperform a simple minimal baseline that samples uniformly
from memory. Our conclusions are consistent in a different number of stream
time steps, e.g., 20 to 200, and under several computational budgets. This
suggests that most existing CL methods are particularly too computationally
expensive for realistic budgeted deployment. Code for this project is available
at: https://github.com/drimpossible/BudgetCL.
Related papers
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based Sampling [19.447914903112366]
We propose to use floating point operations and total memory size in Byte as a metric for computational and memory budgets.
To improve a CL method in a limited total budget, we propose adaptive layer freezing that does not update the layers for less informative batches.
In addition, we propose a memory retrieval method that allows the model to learn the same amount of knowledge as using random retrieval in fewer iterations.
arXiv Detail & Related papers (2024-10-19T16:00:00Z) - Continual Learning on a Diet: Learning from Sparsely Labeled Streams Under Constrained Computation [123.4883806344334]
We study a realistic Continual Learning setting where learning algorithms are granted a restricted computational budget per time step while training.
We apply this setting to large-scale semi-supervised Continual Learning scenarios with sparse label rates.
Our extensive analysis and ablations demonstrate that DietCL is stable under a full spectrum of label sparsity, computational budget, and various other ablations.
arXiv Detail & Related papers (2024-04-19T10:10:39Z) - Density Distribution-based Learning Framework for Addressing Online
Continual Learning Challenges [4.715630709185073]
We introduce a density distribution-based learning framework for online Continual Learning.
Our framework achieves superior average accuracy and time-space efficiency.
Our method outperforms popular CL approaches by a significant margin.
arXiv Detail & Related papers (2023-11-22T09:21:28Z) - Online Continual Learning Without the Storage Constraint [67.66235695269839]
We contribute a simple algorithm, which updates a kNN classifier continually along with a fixed, pretrained feature extractor.
It can adapt to rapidly changing streams, has zero stability gap, operates within tiny computational budgets, has low storage requirements by only storing features.
It can outperform existing methods by over 20% in accuracy on two large-scale online continual learning datasets.
arXiv Detail & Related papers (2023-05-16T08:03:07Z) - Does Continual Learning Equally Forget All Parameters? [55.431048995662714]
Distribution shift (e.g., task or domain shift) in continual learning (CL) usually results in catastrophic forgetting of neural networks.
We study which modules in neural networks are more prone to forgetting by investigating their training dynamics during CL.
We propose a more efficient and simpler method that entirely removes the every-step replay and replaces them by only $k$-times of FPF periodically triggered during CL.
arXiv Detail & Related papers (2023-04-09T04:36:24Z) - Real-Time Evaluation in Online Continual Learning: A New Hope [104.53052316526546]
We evaluate current Continual Learning (CL) methods with respect to their computational costs.
A simple baseline outperforms state-of-the-art CL methods under this evaluation.
This surprisingly suggests that the majority of existing CL literature is tailored to a specific class of streams that is not practical.
arXiv Detail & Related papers (2023-02-02T12:21:10Z) - The CLEAR Benchmark: Continual LEArning on Real-World Imagery [77.98377088698984]
Continual learning (CL) is widely regarded as crucial challenge for lifelong AI.
We introduce CLEAR, the first continual image classification benchmark dataset with a natural temporal evolution of visual concepts.
We find that a simple unsupervised pre-training step can already boost state-of-the-art CL algorithms.
arXiv Detail & Related papers (2022-01-17T09:09:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.