Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How

• URL: http://arxiv.org/abs/2311.04898v2
• Date: Fri, 21 Jun 2024 13:09:07 GMT
• Title: Two Complementary Perspectives to Continual Learning: Ask Not Only What to Optimize, But Also How
• Authors: Timm Hess, Tinne Tuytelaars, Gido M. van de Ven,
• Abstract summary: We argue that continual learning strategies should focus not only on the optimization objective, but also on the way this objective is optimized. We perform a series of pre-registered experiments combining replay-approximated joint objectives with gradient projection-based optimization routines. Our conceptual arguments, as well as some of our empirical results, demonstrate the distinctive importance of the optimization trajectory in continual learning.
• Score: 36.33674007527071
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent years have seen considerable progress in the continual training of deep neural networks, predominantly thanks to approaches that add replay or regularization terms to the loss function to approximate the joint loss over all tasks so far. However, we show that even with a perfect approximation to the joint loss, these approaches still suffer from temporary but substantial forgetting when starting to train on a new task. Motivated by this 'stability gap', we propose that continual learning strategies should focus not only on the optimization objective, but also on the way this objective is optimized. While there is some continual learning work that alters the optimization trajectory (e.g., using gradient projection techniques), this line of research is positioned as alternative to improving the optimization objective, while we argue it should be complementary. In search of empirical support for our proposition, we perform a series of pre-registered experiments combining replay-approximated joint objectives with gradient projection-based optimization routines. However, this first experimental attempt fails to show clear and consistent benefits. Nevertheless, our conceptual arguments, as well as some of our empirical results, demonstrate the distinctive importance of the optimization trajectory in continual learning, thereby opening up a new direction for continual learning research.

Related papers

• Preference-Guided Reinforcement Learning for Efficient Exploration [7.83845308102632]
  We introduce LOPE: Learning Online with trajectory Preference guidancE, an end-to-end preference-guided RL framework. Our intuition is that LOPE directly adjusts the focus of online exploration by considering human feedback as guidance. LOPE outperforms several state-of-the-art methods regarding convergence rate and overall performance.
  arXiv  Detail & Related papers  (2024-07-09T02:11:12Z)
• Improving Forward Compatibility in Class Incremental Learning by Increasing Representation Rank and Feature Richness [3.0620294646308754]
  We introduce an effective-Rank based Feature Richness enhancement (RFR) method, designed for improving forward compatibility. Our results demonstrate the effectiveness of our approach in enhancing novel-task performance while mitigating catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-03-22T11:14:30Z)
• From Function to Distribution Modeling: A PAC-Generative Approach to Offline Optimization [30.689032197123755]
  This paper considers the problem of offline optimization, where the objective function is unknown except for a collection of offline" data examples. Instead of learning and then optimizing the unknown objective function, we take on a less intuitive but more direct view that optimization can be thought of as a process of sampling from a generative model.
  arXiv  Detail & Related papers  (2024-01-04T01:32:50Z)
• Investigation into the Training Dynamics of Learned Optimizers [0.0]
  We look at the concept of learneds as a way to accelerate the optimization process by replacing traditional, hand-crafted algorithms with meta-learned functions. Our work examines their optimization from the perspective of network architecture symmetries and update parameters. We identify several key insights that demonstrate how each approach can benefit from the strengths of the other.
  arXiv  Detail & Related papers  (2023-12-12T11:18:43Z)
• Gradient constrained sharpness-aware prompt learning for vision-language models [99.74832984957025]
  This paper targets a novel trade-off problem in generalizable prompt learning for vision-language models (VLM) By analyzing the loss landscapes of the state-of-the-art method and vanilla Sharpness-aware Minimization (SAM) based method, we conclude that the trade-off performance correlates to both loss value and loss sharpness. We propose a novel SAM-based method for prompt learning, denoted as Gradient Constrained Sharpness-aware Context Optimization (GCSCoOp)
  arXiv  Detail & Related papers  (2023-09-14T17:13:54Z)
• Few-shot Quality-Diversity Optimization [50.337225556491774]
  Quality-Diversity (QD) optimization has been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning. We show that, given examples from a task distribution, information about the paths taken by optimization in parameter space can be leveraged to build a prior population, which when used to initialize QD methods in unseen environments, allows for few-shot adaptation. Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimization in these environments.
  arXiv  Detail & Related papers  (2021-09-14T17:12:20Z)
• Latent-Optimized Adversarial Neural Transfer for Sarcasm Detection [50.29565896287595]
  We apply transfer learning to exploit common datasets for sarcasm detection. We propose a generalized latent optimization strategy that allows different losses to accommodate each other. In particular, we achieve 10.02% absolute performance gain over the previous state of the art on the iSarcasm dataset.
  arXiv  Detail & Related papers  (2021-04-19T13:07:52Z)
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint [35.5156045701898]
  We provide a novel viewpoint of regularization-based continual learning by formulating it as a second-order Taylor approximation of the loss function of each task. Based on this viewpoint, we study the optimization aspects (i.e., convergence) as well as generalization properties (i.e., finite-sample guarantees) of regularization-based continual learning.
  arXiv  Detail & Related papers  (2020-06-19T06:08:40Z)
• Reparameterized Variational Divergence Minimization for Stable Imitation [57.06909373038396]
  We study the extent to which variations in the choice of probabilistic divergence may yield more performant ILO algorithms. We contribute a re parameterization trick for adversarial imitation learning to alleviate the challenges of the promising $f$-divergence minimization framework. Empirically, we demonstrate that our design choices allow for ILO algorithms that outperform baseline approaches and more closely match expert performance in low-dimensional continuous-control tasks.
  arXiv  Detail & Related papers  (2020-06-18T19:04:09Z)
• Disentangling Adaptive Gradient Methods from Learning Rates [65.0397050979662]
  We take a deeper look at how adaptive gradient methods interact with the learning rate schedule. We introduce a "grafting" experiment which decouples an update's magnitude from its direction. We present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods.
  arXiv  Detail & Related papers  (2020-02-26T21:42:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.