Integrating Present and Past in Unsupervised Continual Learning

• URL: http://arxiv.org/abs/2404.19132v2
• Date: Mon, 12 Aug 2024 10:00:17 GMT
• Title: Integrating Present and Past in Unsupervised Continual Learning
• Authors: Yipeng Zhang, Laurent Charlin, Richard Zemel, Mengye Ren,
• Abstract summary: We formulate a unifying framework for unsupervised continual learning (UCL) We show that many existing UCL approaches overlook cross-task consolidation and try to balance plasticity and stability in a shared embedding space.
• Score: 28.208585464074176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We formulate a unifying framework for unsupervised continual learning (UCL), which disentangles learning objectives that are specific to the present and the past data, encompassing stability, plasticity, and cross-task consolidation. The framework reveals that many existing UCL approaches overlook cross-task consolidation and try to balance plasticity and stability in a shared embedding space. This results in worse performance due to a lack of within-task data diversity and reduced effectiveness in learning the current task. Our method, Osiris, which explicitly optimizes all three objectives on separate embedding spaces, achieves state-of-the-art performance on all benchmarks, including two novel benchmarks proposed in this paper featuring semantically structured task sequences. Compared to standard benchmarks, these two structured benchmarks more closely resemble visual signals received by humans and animals when navigating real-world environments. Finally, we show some preliminary evidence that continual models can benefit from such realistic learning scenarios.

Related papers

• Fully Differentiable Bidirectional Dual-Task Synergistic Learning for Semi-Supervised 3D Medical Image Segmentation [3.9950415168730107]
  Semi-supervised learning relaxes the need of large pixel-wise labeled datasets for image segmentation by leveraging unlabeled data.<n>We propose a fully Differentiable Bidirectional Synergistic Learning (DBiSL) framework, which seamlessly integrates and enhances four critical SSL components.
  arXiv  Detail & Related papers  (2026-02-10T03:44:24Z)
• Understanding Catastrophic Interference: On the Identifibility of Latent Representations [67.05452287233122]
  Catastrophic interference, also known as catastrophic forgetting, is a fundamental challenge in machine learning.<n>We propose a novel theoretical framework that formulates catastrophic interference as an identification problem.<n>Our approach provides both theoretical guarantees and practical performance improvements across both synthetic and benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-27T00:53:32Z)
• CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similarity [20.349897901019574]
  We introduce CoDiEmb, a unified framework for training unified text embeddings.<n>CoDiEmb integrates three key innovations for effective joint optimization.<n>Our results and analysis demonstrate that the framework mitigates cross-task trade-offs.
  arXiv  Detail & Related papers  (2025-08-15T12:46:35Z)
• Serial Over Parallel: Learning Continual Unification for Multi-Modal Visual Object Tracking and Benchmarking [35.3869415482475]
  Existing practices mix all data sensor types in a single training procedure, structuring a parallel paradigm from the data-centric perspective.<n>The absence of a unified benchmark where all types of data coexist forces evaluations on separated benchmarks, causing textitinconsistency between training and testing.<n>A unified benchmark, coined as UniBench300, is introduced to bridge the inconsistency by incorporating multiple task data, reducing inference passes from three to one and cutting time consumption by 27%.
  arXiv  Detail & Related papers  (2025-08-14T13:54:04Z)
• Multi-level Collaborative Distillation Meets Global Workspace Model: A Unified Framework for OCIL [38.72433556055473]
  Online Class-Incremental Learning (OCIL) enables models to learn continuously from non-i.i.d. data streams.<n>OCIL faces two key challenges: maintaining model stability under strict memory constraints and ensuring adaptability to new tasks.<n>We propose a novel approach that enhances ensemble learning through a Global Workspace Model (GWM)
  arXiv  Detail & Related papers  (2025-08-12T06:52:33Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Segment Concealed Objects with Incomplete Supervision [63.637733655439334]
  Incompletely-Supervised Concealed Object (ISCOS) involves segmenting objects that seamlessly blend into their surrounding environments.<n>This task remains highly challenging due to the limited supervision provided by the incompletely annotated training data.<n>In this paper, we introduce the first unified method for ISCOS to address these challenges.
  arXiv  Detail & Related papers  (2025-06-10T16:25:15Z)
• A Controllable Examination for Long-Context Language Models [62.845852724511964]
  This study introduces $textbfLongBioBench, a benchmark for evaluating long-context language models.<n>We show that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results.<n>Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence.
  arXiv  Detail & Related papers  (2025-06-03T14:23:06Z)
• Modeling Ranking Properties with In-Context Learning [13.34397013426643]
  We propose an in-context learning (ICL) approach that eliminates the need for task-specific training for each ranking scenario and dataset.<n>Our method relies on a small number of example rankings that demonstrate the desired trade-offs between objectives for past queries similar to the current input.
  arXiv  Detail & Related papers  (2025-05-23T10:58:22Z)
• Underlying Semantic Diffusion for Effective and Efficient In-Context Learning [113.4003355229632]
  Underlying Semantic Diffusion (US-Diffusion) is an enhanced diffusion model that boosts underlying semantics learning, computational efficiency, and in-context learning capabilities. We present a Feedback-Aided Learning (FAL) framework, which leverages feedback signals to guide the model in capturing semantic details. We also propose a plug-and-play Efficient Sampling Strategy (ESS) for dense sampling at time steps with high-noise levels.
  arXiv  Detail & Related papers  (2025-03-06T03:06:22Z)
• Temporal-Difference Variational Continual Learning [89.32940051152782]
  A crucial capability of Machine Learning models in real-world applications is the ability to continuously learn new tasks. In Continual Learning settings, models often struggle to balance learning new tasks with retaining previous knowledge. We propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations.
  arXiv  Detail & Related papers  (2024-10-10T10:58:41Z)
• Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss [3.435381469869212]
  This paper presents an innovative regression framework for Sentence-BERT STS tasks. It proposes two simple yet effective loss functions: Translated ReLU and Smooth K2 Loss. Experimental results demonstrate that our method achieves convincing performance across seven established STS benchmarks.
  arXiv  Detail & Related papers  (2024-06-08T02:52:43Z)
• Improving Data-aware and Parameter-aware Robustness for Continual Learning [3.480626767752489]
  This paper analyzes that this insufficiency arises from the ineffective handling of outliers. We propose a Robust Continual Learning (RCL) method to address this issue. The proposed method effectively maintains robustness and achieves new state-of-the-art (SOTA) results.
  arXiv  Detail & Related papers  (2024-05-27T11:21:26Z)
• Dynamic Sub-graph Distillation for Robust Semi-supervised Continual Learning [52.046037471678005]
  We focus on semi-supervised continual learning (SSCL), where the model progressively learns from partially labeled data with unknown categories. We propose a novel approach called Dynamic Sub-Graph Distillation (DSGD) for semi-supervised continual learning.
  arXiv  Detail & Related papers  (2023-12-27T04:40:12Z)
• BAL: Balancing Diversity and Novelty for Active Learning [53.289700543331925]
  We introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data. Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%.
  arXiv  Detail & Related papers  (2023-12-26T08:14:46Z)
• Synergistic Anchored Contrastive Pre-training for Few-Shot Relation Extraction [4.7220779071424985]
  Few-shot Relation Extraction (FSRE) aims to extract facts from a sparse set of labeled corpora. Recent studies have shown promising results in FSRE by employing Pre-trained Language Models. We introduce a novel synergistic anchored contrastive pre-training framework.
  arXiv  Detail & Related papers  (2023-12-19T10:16:24Z)
• New metrics for analyzing continual learners [27.868967961503962]
  Continual Learning (CL) poses challenges to standard learning algorithms. This stability-plasticity dilemma remains central to CL and multiple metrics have been proposed to adequately measure stability and plasticity separately. We propose new metrics that account for the task's increasing difficulty.
  arXiv  Detail & Related papers  (2023-09-01T13:53:33Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• Multiband VAE: Latent Space Partitioning for Knowledge Consolidation in Continual Learning [14.226973149346883]
  Acquiring knowledge about new data samples without forgetting previous ones is a critical problem of continual learning. We propose a new method for unsupervised continual knowledge consolidation in generative models that relies on the partitioning of Variational Autoencoder's latent space. On top of the standard continual learning evaluation benchmarks, we evaluate our method on a new knowledge consolidation scenario and show that the proposed approach outperforms state-of-the-art by up to twofold.
  arXiv  Detail & Related papers  (2021-06-23T06:58:40Z)
• 3D Human Action Representation Learning via Cross-View Consistency Pursuit [52.19199260960558]
  We propose a Cross-view Contrastive Learning framework for unsupervised 3D skeleton-based action Representation (CrosSCLR) CrosSCLR consists of both single-view contrastive learning (SkeletonCLR) and cross-view consistent knowledge mining (CVC-KM) modules, integrated in a collaborative learning manner.
  arXiv  Detail & Related papers  (2021-04-29T16:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.