RepCL: Exploring Effective Representation for Continual Text Classification

• URL: http://arxiv.org/abs/2305.07289v1
• Date: Fri, 12 May 2023 07:32:00 GMT
• Title: RepCL: Exploring Effective Representation for Continual Text Classification
• Authors: Yifan Song, Peiyi Wang, Dawei Zhu, Tianyu Liu, Zhifang Sui, Sujian Li
• Abstract summary: We focus on continual text classification under the class-incremental setting. Recent CL studies find that the representations learned in one task may not be effective for other tasks. We propose a novel replay-based continual text classification method, RepCL.
• Score: 34.33543812253366
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning (CL) aims to constantly learn new knowledge over time while avoiding catastrophic forgetting on old tasks. In this work, we focus on continual text classification under the class-incremental setting. Recent CL studies find that the representations learned in one task may not be effective for other tasks, namely representation bias problem. For the first time we formally analyze representation bias from an information bottleneck perspective and suggest that exploiting representations with more class-relevant information could alleviate the bias. To this end, we propose a novel replay-based continual text classification method, RepCL. Our approach utilizes contrastive and generative representation learning objectives to capture more class-relevant features. In addition, RepCL introduces an adversarial replay strategy to alleviate the overfitting problem of replay. Experiments demonstrate that RepCL effectively alleviates forgetting and achieves state-of-the-art performance on three text classification tasks.

Related papers

• Replay Consolidation with Label Propagation for Continual Object Detection [7.454468349023651]
  Continual Learning for Object Detection poses additional difficulties compared to CL for Classification. In CLOD, images from previous tasks may contain unknown classes that could reappear labeled in future tasks. We propose a novel technique to solve CLOD called Replay Consolidation with Label Propagation for Object Detection.
  arXiv  Detail & Related papers  (2024-09-09T14:16:27Z)
• InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective [44.961805748830066]
  We focus on continual text classification under the class-incremental setting. Recent studies have identified the severe performance decrease on analogous classes as a key factor for forgetting. We propose a novel replay-based continual text classification method, InfoCL.
  arXiv  Detail & Related papers  (2023-10-10T07:00:13Z)
• Three ways to improve feature alignment for open vocabulary detection [88.65076922242184]
  Key problem in zero-shot open vocabulary detection is how to align visual and text features, so that the detector performs well on unseen classes. Previous approaches train the feature pyramid and detection head from scratch, which breaks the vision-text feature alignment established during pretraining. We propose three methods to alleviate these issues. Firstly, a simple scheme is used to augment the text embeddings which prevents overfitting to a small number of classes seen during training. Secondly, the feature pyramid network and the detection head are modified to include trainable shortcuts. Finally, a self-training approach is used to leverage a larger corpus of
  arXiv  Detail & Related papers  (2023-03-23T17:59:53Z)
• Detachedly Learn a Classifier for Class-Incremental Learning [11.865788374587734]
  We present an analysis that the failure of vanilla experience replay (ER) comes from unnecessary re-learning of previous tasks and incompetence to distinguish current task from the previous ones. We propose a novel replay strategy task-aware experience replay. Experimental results show our method outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-02-23T01:35:44Z)
• Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Models [108.13378788663196]
  We propose Subspace Prompt Tuning (SubPT) to project the gradients in back-propagation onto the low-rank subspace spanned by the early-stage gradient flow eigenvectors during the entire training process. We equip CoOp with Novel Learner Feature (NFL) to enhance the generalization ability of the learned prompts onto novel categories beyond the training set.
  arXiv  Detail & Related papers  (2022-11-04T02:06:22Z)
• New Intent Discovery with Pre-training and Contrastive Learning [21.25371293641141]
  New intent discovery aims to uncover novel intent categories from user utterances to expand the set of supported intent classes. Existing approaches typically rely on a large amount of labeled utterances. We propose a new contrastive loss to exploit self-supervisory signals in unlabeled data for clustering.
  arXiv  Detail & Related papers  (2022-05-25T17:07:25Z)
• Co$^2$L: Contrastive Continual Learning [69.46643497220586]
  Recent breakthroughs in self-supervised learning show that such algorithms learn visual representations that can be transferred better to unseen tasks. We propose a rehearsal-based continual learning algorithm that focuses on continually learning and maintaining transferable representations.
  arXiv  Detail & Related papers  (2021-06-28T06:14:38Z)
• Reducing Representation Drift in Online Continual Learning [87.71558506591937]
  We study the online continual learning paradigm, where agents must learn from a changing distribution with constrained memory and compute. In this work we instead focus on the change in representations of previously observed data due to the introduction of previously unobserved class samples in the incoming data stream.
  arXiv  Detail & Related papers  (2021-04-11T15:19:30Z)
• Revisiting Deep Local Descriptor for Improved Few-Shot Classification [56.74552164206737]
  We show how one can improve the quality of embeddings by leveraging textbfDense textbfClassification and textbfAttentive textbfPooling. We suggest to pool feature maps by applying attentive pooling instead of the widely used global average pooling (GAP) to prepare embeddings for few-shot classification.
  arXiv  Detail & Related papers  (2021-03-30T00:48:28Z)
• On the Exploration of Incremental Learning for Fine-grained Image Retrieval [45.48333682748607]
  We consider the problem of fine-grained image retrieval in an incremental setting, when new categories are added over time. We propose an incremental learning method to mitigate retrieval performance degradation caused by the forgetting issue. Our method effectively mitigates the catastrophic forgetting on the original classes while achieving high performance on the new classes.
  arXiv  Detail & Related papers  (2020-10-15T21:07:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.