Less is More: Rethinking State-of-the-art Continual Relation Extraction Models with a Frustratingly Easy but Effective Approach

• URL: http://arxiv.org/abs/2209.00243v1
• Date: Thu, 1 Sep 2022 06:08:07 GMT
• Title: Less is More: Rethinking State-of-the-art Continual Relation Extraction Models with a Frustratingly Easy but Effective Approach
• Authors: Peiyi Wang, Yifan Song, Tianyu Liu, Rundong Gao, Binghuai Lin, Yunbo Cao and Zhifang Sui
• Abstract summary: Continual relation extraction (CRE) requires the model to continually learn new relations from class-incremental data streams. We propose a Frustratingly easy but Effective Approach (FEA) method with two learning stages for CRE.
• Score: 35.377756110634515
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual relation extraction (CRE) requires the model to continually learn new relations from class-incremental data streams. In this paper, we propose a Frustratingly easy but Effective Approach (FEA) method with two learning stages for CRE: 1) Fast Adaption (FA) warms up the model with only new data. 2) Balanced Tuning (BT) finetunes the model on the balanced memory data. Despite its simplicity, FEA achieves comparable (on TACRED or superior (on FewRel) performance compared with the state-of-the-art baselines. With careful examinations, we find that the data imbalance between new and old relations leads to a skewed decision boundary in the head classifiers over the pretrained encoders, thus hurting the overall performance. In FEA, the FA stage unleashes the potential of memory data for the subsequent finetuning, while the BT stage helps establish a more balanced decision boundary. With a unified view, we find that two strong CRE baselines can be subsumed into the proposed training pipeline. The success of FEA also provides actionable insights and suggestions for future model designing in CRE.

Related papers

• PairCFR: Enhancing Model Training on Paired Counterfactually Augmented Data through Contrastive Learning [49.60634126342945]
  Counterfactually Augmented Data (CAD) involves creating new data samples by applying minimal yet sufficient modifications to flip the label of existing data samples to other classes. Recent research reveals that training with CAD may lead models to overly focus on modified features while ignoring other important contextual information. We employ contrastive learning to promote global feature alignment in addition to learning counterfactual clues.
  arXiv  Detail & Related papers  (2024-06-09T07:29:55Z)
• Rationale-Enhanced Language Models are Better Continual Relation Learners [29.311298089285753]
  Continual relation extraction (CRE) aims to solve the problem of catastrophic forgetting when learning a sequence of newly emerging relations. Recent CRE studies have found that catastrophic forgetting arises from the model's lack of robustness against future analogous relations.
  arXiv  Detail & Related papers  (2023-10-10T11:50:27Z)
• Alleviating the Effect of Data Imbalance on Adversarial Training [26.36714114672729]
  We study adversarial training on datasets that obey the long-tailed distribution. We propose a new adversarial training framework -- Re-balancing Adversarial Training (REAT)
  arXiv  Detail & Related papers  (2023-07-14T07:01:48Z)
• Enhancing Continual Relation Extraction via Classifier Decomposition [30.88081408988638]
  Continual relation extraction models aim at handling emerging new relations while avoiding forgetting old ones in the streaming data. Most models only adopt a vanilla strategy when models first learn representations of new relations. We propose a simple yet effective classifier decomposition framework that splits the last FFN layer into separated previous and current classifiers.
  arXiv  Detail & Related papers  (2023-05-08T11:29:33Z)
• CORSD: Class-Oriented Relational Self Distillation [16.11986532440837]
  Knowledge distillation conducts an effective model compression method while holding some limitations. We propose a novel training framework named Class-Oriented Self Distillation (CORSD) to address the limitations.
  arXiv  Detail & Related papers  (2023-04-28T16:00:31Z)
• Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation [45.87125587600661]
  Continual relation extraction (CRE) aims to continually learn new relations from a class-incremental data stream. CRE model usually suffers from catastrophic forgetting problem, i.e., the performance of old relations seriously degrades when the model learns new relations. To address this issue, we encourage the model to learn more precise and robust representations through a simple yet effective adversarial class augmentation mechanism.
  arXiv  Detail & Related papers  (2022-10-10T08:50:48Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Learning of Visual Relations: The Devil is in the Tails [59.737494875502215]
  Visual relation learning is a long-tailed problem, due to the nature of joint reasoning about groups of objects. In this paper, we explore an alternative hypothesis, denoted the Devil is in the Tails. Under this hypothesis, better performance is achieved by keeping the model simple but improving its ability to cope with long-tailed distributions.
  arXiv  Detail & Related papers  (2021-08-22T08:59:35Z)
• Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data [49.378860065474875]
  We identify failure modes of SOTA relation extraction (RE) models trained on TACRED. By adding some of the challenge data as training examples, the performance of the model improves.
  arXiv  Detail & Related papers  (2020-10-07T21:17:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.