Deep Bayesian Unsupervised Lifelong Learning

• URL: http://arxiv.org/abs/2106.07035v1
• Date: Sun, 13 Jun 2021 16:24:44 GMT
• Title: Deep Bayesian Unsupervised Lifelong Learning
• Authors: Tingting Zhao, Zifeng Wang, Aria Masoomi, Jennifer Dy
• Abstract summary: We focus on resolving challenges in Unsupervised Lifelong Learning (ULL) with streaming unlabelled data. We develop a fully Bayesian inference framework for ULL with a novel end-to-end Deep Bayesian Unsupervised Lifelong Learning (DBULL) algorithm. To efficiently maintain past knowledge, we develop a novel knowledge preservation mechanism via sufficient statistics of the latent representation for raw data.
• Score: 3.4827140757744908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lifelong Learning (LL) refers to the ability to continually learn and solve new problems with incremental available information over time while retaining previous knowledge. Much attention has been given lately to Supervised Lifelong Learning (SLL) with a stream of labelled data. In contrast, we focus on resolving challenges in Unsupervised Lifelong Learning (ULL) with streaming unlabelled data when the data distribution and the unknown class labels evolve over time. Bayesian framework is natural to incorporate past knowledge and sequentially update the belief with new data. We develop a fully Bayesian inference framework for ULL with a novel end-to-end Deep Bayesian Unsupervised Lifelong Learning (DBULL) algorithm, which can progressively discover new clusters without forgetting the past with unlabelled data while learning latent representations. To efficiently maintain past knowledge, we develop a novel knowledge preservation mechanism via sufficient statistics of the latent representation for raw data. To detect the potential new clusters on the fly, we develop an automatic cluster discovery and redundancy removal strategy in our inference inspired by Nonparametric Bayesian statistics techniques. We demonstrate the effectiveness of our approach using image and text corpora benchmark datasets in both LL and batch settings.

Related papers

• Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models [79.28821338925947]
  Domain-Class Incremental Learning is a realistic but challenging continual learning scenario. To handle these diverse tasks, pre-trained Vision-Language Models (VLMs) are introduced for their strong generalizability. This incurs a new problem: the knowledge encoded in the pre-trained VLMs may be disturbed when adapting to new tasks, compromising their inherent zero-shot ability. Existing methods tackle it by tuning VLMs with knowledge distillation on extra datasets, which demands heavy overhead. We propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of
  arXiv  Detail & Related papers  (2024-07-07T12:19:37Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• COOLer: Class-Incremental Learning for Appearance-Based Multiple Object Tracking [32.47215340215641]
  This paper extends the scope of continual learning research to class-incremental learning for multiple object tracking (MOT) Previous solutions for continual learning of object detectors do not address the data association stage of appearance-based trackers. We introduce COOLer, a COntrastive- and cOntinual-Learning-based tracker, which incrementally learns to track new categories while preserving past knowledge.
  arXiv  Detail & Related papers  (2023-10-04T17:49:48Z)
• VERSE: Virtual-Gradient Aware Streaming Lifelong Learning with Anytime Inference [36.61783715563126]
  Streaming lifelong learning is a challenging setting of lifelong learning with the goal of continuous learning without forgetting. We introduce a novel approach to lifelong learning, which is streaming (observes each training example only once) We propose a novel emphvirtual gradients based approach for continual representation learning which adapts to each new example while also generalizing well on past data to prevent catastrophic forgetting.
  arXiv  Detail & Related papers  (2023-09-15T07:54:49Z)
• CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation [128.00940554196976]
  Vision-Language Continual Pretraining (VLCP) has shown impressive results on diverse downstream tasks by offline training on large-scale datasets. To support the study of Vision-Language Continual Pretraining (VLCP), we first contribute a comprehensive and unified benchmark dataset P9D. The data from each industry as an independent task supports continual learning and conforms to the real-world long-tail nature to simulate pretraining on web data.
  arXiv  Detail & Related papers  (2023-08-14T13:53:18Z)
• Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning [133.39254981496146]
  Class-incremental learning (CIL) suffers from the notorious dilemma between learning newly added classes and preserving previously learned class knowledge. We propose to leverage "free" external unlabeled data querying in continual learning. We show queried unlabeled data can continue to benefit, and seamlessly extend CIL-QUD into its robustified versions.
  arXiv  Detail & Related papers  (2022-06-15T22:53:23Z)
• Contrastive Learning with Boosted Memorization [36.957895270908324]
  Self-supervised learning has achieved a great success in the representation learning of visual and textual data. Recent attempts to consider self-supervised long-tailed learning are made by rebalancing in the loss perspective or the model perspective. We propose a novel Boosted Contrastive Learning (BCL) method to enhance the long-tailed learning in the label-unaware context.
  arXiv  Detail & Related papers  (2022-05-25T11:54:22Z)
• Learning to Predict Gradients for Semi-Supervised Continual Learning [36.715712711431856]
  Key challenge for machine intelligence is to learn new visual concepts without forgetting the previously acquired knowledge. There is a gap between existing supervised continual learning and human-like intelligence, where human is able to learn from both labeled and unlabeled data. We formulate a new semi-supervised continual learning method, which can be generically applied to existing continual learning models.
  arXiv  Detail & Related papers  (2022-01-23T06:45:47Z)
• Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data [101.6195176510611]
  "Online" continual learning enables evaluating both information retention and online learning efficacy. In online continual learning, each incoming small batch of data is first used for testing and then added to the training set, making the problem truly online. We introduce a new benchmark for online continual visual learning that exhibits large scale and natural distribution shifts.
  arXiv  Detail & Related papers  (2021-08-20T06:17:20Z)
• Continual Learning From Unlabeled Data Via Deep Clustering [7.704949298975352]
  Continual learning aims to learn new tasks incrementally using less computation and memory resources instead of retraining the model from scratch whenever new task arrives. We introduce a new framework to make continual learning feasible in unsupervised mode by using pseudo label obtained from cluster assignments to update model.
  arXiv  Detail & Related papers  (2021-04-14T23:46:17Z)
• ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-supervised Continual Learning [52.831894583501395]
  Continual learning assumes the incoming data are fully labeled, which might not be applicable in real applications. We propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN) We show ORDisCo achieves significant performance improvement on various semi-supervised learning benchmark datasets for SSCL.
  arXiv  Detail & Related papers  (2021-01-02T09:04:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.