Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks

• URL: http://arxiv.org/abs/2605.05358v1
• Date: Wed, 06 May 2026 18:36:32 GMT
• Title: Balancing Stability and Plasticity in Sequentially Trained Early-Exiting Neural Networks
• Abstract summary: Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers.<n>We propose two alternative approaches that operate at different levels of the model.<n> Experiments on standard benchmarks show that our approaches consistently improve early-exit performance.
• Score: 11.704570155438887
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Early-exiting neural networks enable adaptive inference by allowing inputs to exit at intermediate classifiers, reducing computation for easy samples while maintaining high accuracy. In practice, exits can be trained sequentially by incrementally adding them to a shared backbone; however, this sequential training can cause newly introduced exits to interfere with previously learned ones, degrading the performance of earlier classifiers. We address this problem by retaining the knowledge embedded in existing exits while allowing new ones to specialize. We propose two alternative approaches that operate at different levels of the model. The first constrains learning by protecting parameters that are important for previously trained exits, while the second preserves the output distributions of earlier exits as the network adapts. These alternatives directly reflect the stability-plasticity trade-off studied in continual learning. Accordingly, we leverage \textit{Elastic Weight Consolidation} to constrain critical weights and \textit{Learning without Forgetting} to preserve output distributions. Experiments on standard benchmarks show that our approaches consistently improve early-exit performance, achieving higher accuracy over existing sequential training methods and significant performance speedups at low computational budgets.

Related papers

• PriFT: Prior-Support Guided Supervised Fine-Tuning [74.65198014829393]
  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation.<n>SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution.<n>We propose PriFT, which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T12:14:06Z)
• Task Switching Without Forgetting via Proximal Decoupling [43.88767637201092]
  In continual learning, the primary challenge is to learn new information without forgetting old knowledge.<n>A common solution addresses this trade-off through regularization, penalizing changes to parameters critical for previous tasks.<n>We propose a different approach that separates task learning from stability enforcement via operator splitting.
  arXiv  Detail & Related papers  (2026-04-20T21:28:13Z)
• Online Bayesian Imbalanced Learning with Bregman-Calibrated Deep Networks [0.7106986689736825]
  We present textitOnline Bayesian Imbalanced Learning (OBIL), a principled framework that decouples likelihood-ratio estimation from class-prior assumptions.<n>Our approach builds on the established connection between Bregman divergences and proper scoring rules to show that deep networks trained with such losses produce posterior probability estimates.<n>We prove that these likelihood-ratio estimates remain valid under arbitrary changes in class priors and cost structures, requiring only a threshold adjustment for optimal Bayes decisions.
  arXiv  Detail & Related papers  (2026-02-08T21:23:00Z)
• Boosted Training of Lightweight Early Exits for Optimizing CNN Image Classification Inference [47.027290803102666]
  We introduce a sequential training approach that aligns branch training with inference-time data distributions.<n>Experiments on the CINIC-10 dataset with a ResNet18 backbone demonstrate that BTS-EE consistently outperforms non-boosted training.<n>These results offer practical efficiency gains for applications such as industrial inspection, embedded vision, and UAV-based monitoring.
  arXiv  Detail & Related papers  (2025-09-10T06:47:49Z)
• Training of Spiking Neural Networks with Expectation-Propagation [9.24888258922809]
  We propose a unifying message-passing framework for training spiking neural networks (SNNs)<n>Our gradient-free method is capable of learning the marginal distributions of network parameters and simultaneously marginalizes parameters, such as the outputs of hidden layers.
  arXiv  Detail & Related papers  (2025-06-30T11:59:56Z)
• Instance-dependent Early Stopping [57.912273923450726]
  We propose an Instance-dependent Early Stopping (IES) method that adapts the early stopping mechanism from the entire training set to the instance level.<n>IES considers an instance as mastered if the second-order differences of its loss value remain within a small range around zero.<n>IES can reduce backpropagation instances by 10%-50% while maintaining or even slightly improving the test accuracy and transfer learning performance of a model.
  arXiv  Detail & Related papers  (2025-02-11T13:34:09Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Toward Understanding BERT-Like Pre-Training for DNA Foundation Models [78.48760388079523]
  Existing pre-training methods for DNA sequences rely on direct adoptions of BERT pre-training from NLP. We introduce a novel approach called RandomMask, which gradually increases the task difficulty of BERT-like pre-training by continuously expanding its mask boundary. RandomMask achieves a staggering 68.16% in Matthew's correlation coefficient for Epigenetic Mark Prediction, a groundbreaking increase of 19.85% over the baseline.
  arXiv  Detail & Related papers  (2023-10-11T16:40:57Z)
• TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization [89.54947228958494]
  This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework. TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
  arXiv  Detail & Related papers  (2023-03-20T14:12:55Z)
• Two-phase Pseudo Label Densification for Self-training based Domain Adaptation [93.03265290594278]
  We propose a novel Two-phase Pseudo Label Densification framework, referred to as TPLD. In the first phase, we use sliding window voting to propagate the confident predictions, utilizing intrinsic spatial-correlations in the images. In the second phase, we perform a confidence-based easy-hard classification. To ease the training process and avoid noisy predictions, we introduce the bootstrapping mechanism to the original self-training loss.
  arXiv  Detail & Related papers  (2020-12-09T02:35:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.