Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning

• URL: http://arxiv.org/abs/2505.14125v2
• Date: Tue, 04 Nov 2025 14:37:03 GMT
• Title: Contrastive Consolidation of Top-Down Modulations Achieves Sparsely Supervised Continual Learning
• Authors: Viet Anh Khoa Tran, Emre Neftci, Willem A. M. Wybo,
• Abstract summary: We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex.<n>TMCL integrates top-down information continually and without supervision.<n>Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches.
• Score: 5.3151394920162645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Biological brains learn continually from a stream of unlabeled data, while integrating specialized information from sparsely labeled examples without compromising their ability to generalize. Meanwhile, machine learning methods are susceptible to catastrophic forgetting in this natural learning setting, as supervised specialist fine-tuning degrades performance on the original task. We introduce task-modulated contrastive learning (TMCL), which takes inspiration from the biophysical machinery in the neocortex, using predictive coding principles to integrate top-down information continually and without supervision. We follow the idea that these principles build a view-invariant representation space, and that this can be implemented using a contrastive loss. Then, whenever labeled samples of a new class occur, new affine modulations are learned that improve separation of the new class from all others, without affecting feedforward weights. By co-opting the view-invariance learning mechanism, we then train feedforward weights to match the unmodulated representation of a data sample to its modulated counterparts. This introduces modulation invariance into the representation space, and, by also using past modulations, stabilizes it. Our experiments show improvements in both class-incremental and transfer learning over state-of-the-art unsupervised approaches, as well as over comparable supervised approaches, using as few as 1% of available labels. Taken together, our work suggests that top-down modulations play a crucial role in balancing stability and plasticity.

Related papers

• Resilient Class-Incremental Learning: on the Interplay of Drifting, Unlabelled and Imbalanced Data Streams [3.3977968101293956]
  This paper proposes SCIL (Streaming Class-Incremental Learning) to address these challenges.<n>The framework integrates an autoencoder (AE) with a multi-layer perceptron for multi-class prediction, uses a dual-loss strategy (classification and reconstruction) for prediction and new class detection, employs corrected pseudo-labels for online training, manages classes with queues, and applies oversampling to handle imbalance.<n>Our results demonstrate that SCIL outperforms strong baselines and state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-02-10T11:37:39Z)
• Orthogonal Activation with Implicit Group-Aware Bias Learning for Class Imbalance [1.2891210250935148]
  Class imbalance is a common challenge in machine learning and data mining.<n>We propose a novel activation function, named OGAB, designed to alleviate class imbalance in deep learning classifiers.<n>We demonstrate the effectiveness of our solution on both real-world and synthetic imbalanced datasets.
  arXiv  Detail & Related papers  (2025-12-23T03:05:25Z)
• Adaptive Weighted Parameter Fusion with CLIP for Class-Incremental Learning [12.67816343247008]
  Class-incremental Learning enables the model to incrementally absorb knowledge from new classes.<n>When the model optimize with new classes, the knowledge of previous classes is inevitably erased, leading to catastrophic forgetting.
  arXiv  Detail & Related papers  (2025-03-25T09:51:04Z)
• DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning [23.878495627964146]
  Continual learning aims to equip models with the ability to retain previously learned knowledge like a human.<n>Existing methods usually overlook the issue of information leakage caused by the fact that the experiment data have been used in pre-trained models.<n>In this paper, we propose a new LoRA-based rehearsal-free method named DESIRE.
  arXiv  Detail & Related papers  (2024-11-28T13:54:01Z)
• Stochastic Vision Transformers with Wasserstein Distance-Aware Attention [8.407731308079025]
  Self-supervised learning is one of the most promising approaches to acquiring knowledge from limited labeled data. We introduce a new vision transformer that integrates uncertainty and distance awareness into self-supervised learning pipelines. Our proposed method achieves superior accuracy and calibration, surpassing the self-supervised baseline in a wide range of experiments on a variety of datasets.
  arXiv  Detail & Related papers  (2023-11-30T15:53:37Z)
• Learning Invariant Molecular Representation in Latent Discrete Space [52.13724532622099]
  We propose a new framework for learning molecular representations that exhibit invariance and robustness against distribution shifts. Our model achieves stronger generalization against state-of-the-art baselines in the presence of various distribution shifts.
  arXiv  Detail & Related papers  (2023-10-22T04:06:44Z)
• Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations [22.289830907729705]
  Online continual learning (OCL) aims to enable model learning from a non-stationary data stream to continuously acquire new knowledge as well as retain the learnt one. Main challenge comes from the "catastrophic forgetting" issue -- the inability to well remember the learnt knowledge while learning the new ones.
  arXiv  Detail & Related papers  (2022-11-10T05:29:43Z)
• Constructing Balance from Imbalance for Long-tailed Image Recognition [50.6210415377178]
  The imbalance between majority (head) classes and minority (tail) classes severely skews the data-driven deep neural networks. Previous methods tackle with data imbalance from the viewpoints of data distribution, feature space, and model design. We propose a concise paradigm by progressively adjusting label space and dividing the head classes and tail classes. Our proposed model also provides a feature evaluation method and paves the way for long-tailed feature learning.
  arXiv  Detail & Related papers  (2022-08-04T10:22:24Z)
• Automatic Rule Induction for Efficient Semi-Supervised Learning [56.91428251227253]
  Semi-supervised learning has shown promise in allowing NLP models to generalize from small amounts of labeled data. Pretrained transformer models act as black-box correlation engines that are difficult to explain and sometimes behave unreliably. We propose tackling both of these challenges via Automatic Rule Induction (ARI), a simple and general-purpose framework.
  arXiv  Detail & Related papers  (2022-05-18T16:50:20Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• Analyzing Overfitting under Class Imbalance in Neural Networks for Image Segmentation [19.259574003403998]
  In image segmentation neural networks may overfit to the foreground samples from small structures. In this study, we provide new insights on the problem of overfitting under class imbalance by inspecting the network behavior.
  arXiv  Detail & Related papers  (2021-02-20T14:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.