Consistent Representation Learning for Continual Relation Extraction
- URL: http://arxiv.org/abs/2203.02721v1
- Date: Sat, 5 Mar 2022 12:16:34 GMT
- Title: Consistent Representation Learning for Continual Relation Extraction
- Authors: Kang Zhao and Hua Xu and Jiangong Yang and Kai Gao
- Abstract summary: A consistent representation learning method is proposed, which maintains the stability of the relation embedding.
Our method significantly outperforms state-of-the-art baselines and yield strong robustness on the imbalanced dataset.
- Score: 18.694012937149495
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Continual relation extraction (CRE) aims to continuously train a model on
data with new relations while avoiding forgetting old ones. Some previous work
has proved that storing a few typical samples of old relations and replaying
them when learning new relations can effectively avoid forgetting. However,
these memory-based methods tend to overfit the memory samples and perform
poorly on imbalanced datasets. To solve these challenges, a consistent
representation learning method is proposed, which maintains the stability of
the relation embedding by adopting contrastive learning and knowledge
distillation when replaying memory. Specifically, supervised contrastive
learning based on a memory bank is first used to train each new task so that
the model can effectively learn the relation representation. Then, contrastive
replay is conducted of the samples in memory and makes the model retain the
knowledge of historical relations through memory knowledge distillation to
prevent the catastrophic forgetting of the old task. The proposed method can
better learn consistent representations to alleviate forgetting effectively.
Extensive experiments on FewRel and TACRED datasets show that our method
significantly outperforms state-of-the-art baselines and yield strong
robustness on the imbalanced dataset.
Related papers
- Contrastive Continual Learning with Importance Sampling and
Prototype-Instance Relation Distillation [14.25441464051506]
We propose Contrastive Continual Learning via Importance Sampling (CCLIS) to preserve knowledge by recovering previous data distributions.
We also present the Prototype-instance Relation Distillation (PRD) loss, a technique designed to maintain the relationship between prototypes and sample representations.
arXiv Detail & Related papers (2024-03-07T15:47:52Z) - Enhancing Consistency and Mitigating Bias: A Data Replay Approach for
Incremental Learning [100.7407460674153]
Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks.
To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks.
However, it is not expected in practice considering the memory constraint or data privacy issue.
As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
arXiv Detail & Related papers (2024-01-12T12:51:12Z) - Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information.
We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting.
Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
arXiv Detail & Related papers (2023-09-15T17:10:51Z) - Serial Contrastive Knowledge Distillation for Continual Few-shot
Relation Extraction [35.79570854392989]
We propose a new model, namely SCKD, to accomplish the continual few-shot RE task.
Specifically, we design serial knowledge distillation to preserve the prior knowledge from previous models.
Our experiments on two benchmark datasets validate the effectiveness of SCKD for continual few-shot RE.
arXiv Detail & Related papers (2023-05-11T07:25:47Z) - A Memory Transformer Network for Incremental Learning [64.0410375349852]
We study class-incremental learning, a training setup in which new classes of data are observed over time for the model to learn from.
Despite the straightforward problem formulation, the naive application of classification models to class-incremental learning results in the "catastrophic forgetting" of previously seen classes.
One of the most successful existing methods has been the use of a memory of exemplars, which overcomes the issue of catastrophic forgetting by saving a subset of past data into a memory bank and utilizing it to prevent forgetting when training future tasks.
arXiv Detail & Related papers (2022-10-10T08:27:28Z) - Continual Few-shot Relation Learning via Embedding Space Regularization
and Data Augmentation [4.111899441919165]
It is necessary for the model to learn novel relational patterns with very few labeled data while avoiding catastrophic forgetting of previous task knowledge.
We propose a novel method based on embedding space regularization and data augmentation.
Our method generalizes to new few-shot tasks and avoids catastrophic forgetting of previous tasks by enforcing extra constraints on the relational embeddings and by adding extra relevant data in a self-supervised manner.
arXiv Detail & Related papers (2022-03-04T05:19:09Z) - Relational Experience Replay: Continual Learning by Adaptively Tuning
Task-wise Relationship [54.73817402934303]
We propose Experience Continual Replay (ERR), a bi-level learning framework to adaptively tune task-wise to achieve a better stability plasticity' tradeoff.
ERR can consistently improve the performance of all baselines and surpass current state-of-the-art methods.
arXiv Detail & Related papers (2021-12-31T12:05:22Z) - Understanding Catastrophic Forgetting and Remembering in Continual
Learning with Optimal Relevance Mapping [10.970706194360451]
Catastrophic forgetting in neural networks is a significant problem for continual learning.
We introduce Relevance Mapping Networks (RMNs) which are inspired by the Optimal Overlap Hypothesis.
We show that RMNs learn an optimized representational overlap that overcomes the twin problem of catastrophic forgetting and remembering.
arXiv Detail & Related papers (2021-02-22T20:34:00Z) - Relation-Guided Representation Learning [53.60351496449232]
We propose a new representation learning method that explicitly models and leverages sample relations.
Our framework well preserves the relations between samples.
By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
arXiv Detail & Related papers (2020-07-11T10:57:45Z) - Automatic Recall Machines: Internal Replay, Continual Learning and the
Brain [104.38824285741248]
Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity.
We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective.
Instead the implicit memory of learned samples within the assessed model itself is exploited.
arXiv Detail & Related papers (2020-06-22T15:07:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.