Continual Learning in Deep Networks: an Analysis of the Last Layer

• URL: http://arxiv.org/abs/2106.01834v1
• Date: Thu, 3 Jun 2021 13:41:29 GMT
• Title: Continual Learning in Deep Networks: an Analysis of the Last Layer
• Abstract summary: Three factors affect catastrophic forgetting in the output layer: (1) weights modifications, (2) interferences, and (3) projection drift. We show that the best-performing output layer type depends on the data distribution drifts or the amount of data available.
• Score: 7.405588182540833
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study how different output layer types of a deep neural network learn and forget in continual learning settings. We describe the three factors affecting catastrophic forgetting in the output layer: (1) weights modifications, (2) interferences, and (3) projection drift. Our goal is to provide more insights into how different types of output layers can address (1) and (2). We also propose potential solutions and evaluate them on several benchmarks. We show that the best-performing output layer type depends on the data distribution drifts or the amount of data available. In particular, in some cases where a standard linear layer would fail, it is sufficient to change the parametrization and get significantly better performance while still training with SGD. Our results and analysis shed light on the dynamics of the output layer in continual learning scenarios and help select the best-suited output layer for a given scenario.

Related papers

• When Does Sparsity Mitigate the Curse of Depth in LLMs [53.137717161619484]
  We show that sparsity acts as a regulator of variance propagation and thereby improves depth utilization.<n>Our results reveal sparsity as a key yet previously overlooked mechanism for effective depth scaling in large language models.
  arXiv  Detail & Related papers  (2026-03-16T15:04:16Z)
• Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models [51.754991950934375]
  In a pretrained VLM, all layers are engaged by default to make predictions on downstream tasks.<n>We find that intervening on a single layer, such as by zeroing its parameters, can improve the performance on certain tasks.<n>We propose TaLo, a training-free, test-time adaptation method that dynamically identifies and bypasses the most interfering layer for a given task.
  arXiv  Detail & Related papers  (2026-02-01T11:37:05Z)
• LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions [33.23446361483562]
  Pretrained Large Language Models (LLMs) achieve strong performance across a wide range of tasks.<n>LLMs exhibit substantial variability in the various layers' training quality with respect to specific downstream applications.<n>We propose LayerIF to quantify the training quality of individual layers in a principled and task-sensitive manner.
  arXiv  Detail & Related papers  (2025-05-27T10:55:12Z)
• Initialization Matters: On the Benign Overfitting of Two-Layer ReLU CNN with Fully Trainable Layers [20.25049261035324]
  We extend the analysis to two-layer ReLU convolutional neural networks (CNNs) with fully trainable layers. Our results show that the scaling of the output layer is crucial to the training dynamics. In both settings, we provide nearly matching upper and lower bounds on the test errors.
  arXiv  Detail & Related papers  (2024-10-24T20:15:45Z)
• Entropy Guided Extrapolative Decoding to Improve Factuality in Large Language Models [55.45444773200529]
  Large language models (LLMs) exhibit impressive natural language capabilities but suffer from hallucination. Recent work has focused on decoding techniques to improve factuality during inference.
  arXiv  Detail & Related papers  (2024-04-14T19:45:35Z)
• Layer-wise Linear Mode Connectivity [52.6945036534469]
  Averaging neural network parameters is an intuitive method for the knowledge of two independent models. It is most prominently used in federated learning. We analyse the performance of the models that result from averaging single, or groups.
  arXiv  Detail & Related papers  (2023-07-13T09:39:10Z)
• Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond [52.656743602538825]
  Fine-tuning pre-trained code models incurs a large computational cost. We conduct an experimental study to explore what happens to layer-wise pre-trained representations and their encoded code knowledge during fine-tuning. We propose Telly to efficiently fine-tune pre-trained code models via layer freezing.
  arXiv  Detail & Related papers  (2023-04-11T13:34:13Z)
• Surgical Fine-Tuning Improves Adaptation to Distribution Shifts [114.17184775397067]
  A common approach to transfer learning under distribution shift is to fine-tune the last few layers of a pre-trained model. This paper shows that in such settings, selectively fine-tuning a subset of layers matches or outperforms commonly used fine-tuning approaches.
  arXiv  Detail & Related papers  (2022-10-20T17:59:15Z)
• SSMTL++: Revisiting Self-Supervised Multi-Task Learning for Video Anomaly Detection [108.57862846523858]
  We revisit the self-supervised multi-task learning framework, proposing several updates to the original method. We modernize the 3D convolutional backbone by introducing multi-head self-attention modules. In our attempt to further improve the model, we study additional self-supervised learning tasks, such as predicting segmentation maps.
  arXiv  Detail & Related papers  (2022-07-16T19:25:41Z)
• SuperDeConFuse: A Supervised Deep Convolutional Transform based Fusion Framework for Financial Trading Systems [29.411173536818477]
  This work proposes a supervised multi-channel time-series learning framework for financial stock trading. Our approach consists of processing the data channels through separate 1-D convolution layers, then fusing the outputs with a series of fully-connected layers, and finally applying a softmax classification layer. Numerical experiments confirm that the proposed model yields considerably better results than state-of-the-art deep learning techniques for real-world problem of stock trading.
  arXiv  Detail & Related papers  (2020-11-09T11:58:12Z)
• Training with Multi-Layer Embeddings for Model Reduction [0.9046327456472286]
  We introduce a multi-layer embedding training architecture that trains embeddings via a sequence of linear layers. We show that it allows reducing d by 4-8X, with a corresponding improvement in memory footprint, at given model accuracy.
  arXiv  Detail & Related papers  (2020-06-10T02:47:40Z)
• PLLay: Efficient Topological Layer based on Persistence Landscapes [24.222495922671442]
  PLLay is a novel topological layer for general deep learning models based on persistence landscapes. We show differentiability with respect to layer inputs, for a general persistent homology with arbitrary filtration.
  arXiv  Detail & Related papers  (2020-02-07T13:34:22Z)
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference [82.96877371742532]
  We propose a Dynamic Fractional Skipping (DFS) framework for deep networks. DFS hypothesizes layer-wise quantization (to different bitwidths) as intermediate "soft" choices to be made between fully utilizing and skipping a layer. It exploits a layer's expressive power during input-adaptive inference, enabling finer-grained accuracy-computational cost trade-offs.
  arXiv  Detail & Related papers  (2020-01-03T03:12:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.