Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training

• URL: http://arxiv.org/abs/2106.10835v1
• Date: Mon, 21 Jun 2021 03:54:02 GMT
• Title: Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training
• Authors: Tao Chen, Haochen Shi, Liyuan Liu, Siliang Tang, Jian Shao, Zhigang Chen, Yueting Zhuang
• Abstract summary: We propose collaborative adversarial training to improve the data utilization of multi-instance learning (MIL) Since VAT is label-free, we employ the instance-level VAT to recycle instances abandoned by MIL. Our proposed method brings consistent improvements ( 5 absolute AUC score) to the previous state of the art.
• Score: 53.347081723070666
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With recent advances in distantly supervised (DS) relation extraction (RE), considerable attention is attracted to leverage multi-instance learning (MIL) to distill high-quality supervision from the noisy DS. Here, we go beyond label noise and identify the key bottleneck of DS-MIL to be its low data utilization: as high-quality supervision being refined by MIL, MIL abandons a large amount of training instances, which leads to a low data utilization and hinders model training from having abundant supervision. In this paper, we propose collaborative adversarial training to improve the data utilization, which coordinates virtual adversarial training (VAT) and adversarial training (AT) at different levels. Specifically, since VAT is label-free, we employ the instance-level VAT to recycle instances abandoned by MIL. Besides, we deploy AT at the bag-level to unleash the full potential of the high-quality supervision got by MIL. Our proposed method brings consistent improvements (~ 5 absolute AUC score) to the previous state of the art, which verifies the importance of the data utilization issue and the effectiveness of our method.

Related papers

• A few-shot Label Unlearning in Vertical Federated Learning [16.800865928660954]
  This paper addresses the critical challenge of unlearning in Vertical Federated Learning (VFL) We introduce the first approach specifically designed to tackle label unlearning in VFL, focusing on scenarios where the active party aims to mitigate the risk of label leakage. Our method leverages a limited amount of labeled data, utilizing manifold mixup to augment the forward embedding of insufficient data, followed by gradient ascent on the augmented embeddings to erase label information from the models.
  arXiv  Detail & Related papers  (2024-10-14T12:08:12Z)
• Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate [118.37653302885607]
  We present the Modality Integration Rate (MIR), an effective, robust, and generalized metric to indicate the multi-modal pre-training quality of Large Vision Language Models (LVLMs) MIR is indicative about training data selection, training strategy schedule, and model architecture design to get better pre-training results.
  arXiv  Detail & Related papers  (2024-10-09T17:59:04Z)
• ProFeAT: Projected Feature Adversarial Training for Self-Supervised Learning of Robust Representations [35.68752612346952]
  The need for abundant labelled data in supervised Adversarial Training (AT) has prompted the use of Self-Supervised Learning (SSL) techniques with AT. The direct application of existing SSL methods to adversarial training has been sub-optimal due to the increased training complexity of combining SSL with AT. We propose appropriate attack and defense losses at the feature and projector, alongside a combination of weak and strong augmentations for the teacher and student respectively.
  arXiv  Detail & Related papers  (2024-06-09T14:20:46Z)
• PDL: Regularizing Multiple Instance Learning with Progressive Dropout Layers [2.069061136213899]
  Multiple instance learning (MIL) was a weakly supervised learning approach that sought to assign binary class labels to collections of instances known as bags. We present a novel approach in the form of a Progressive Dropout Layer (PDL) to address overfitting and empower the MIL model in uncovering intricate and impactful feature representations.
  arXiv  Detail & Related papers  (2023-08-19T21:20:30Z)
• Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneity [60.791736094073]
  Communication overhead has become one of the major bottlenecks in the distributed training of deep neural networks. We propose a magnitude-driven sparsification scheme, which addresses the non-convergence issue of SIGNSGD. The proposed scheme is validated through experiments on Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2023-02-19T17:42:35Z)
• Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning [10.724516317292924]
  We show how RM can be approached as a multiple instance learning (MIL) problem. We develop new MIL models that are able to capture the time dependencies in labelled trajectories. We demonstrate on a range of RL tasks that our novel MIL models can reconstruct reward functions to a high level of accuracy.
  arXiv  Detail & Related papers  (2022-05-30T18:20:22Z)
• CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction [52.94486705393062]
  We go beyond typical multi-instance learning (MIL) framework and propose a novel contrastive instance learning (CIL) framework. Specifically, we regard the initial MIL as the relational triple encoder and constraint positive pairs against negative pairs for each instance. Experiments demonstrate the effectiveness of our proposed framework, with significant improvements over the previous methods on NYT10, GDS and KBP.
  arXiv  Detail & Related papers  (2021-06-21T04:51:59Z)
• Softmax with Regularization: Better Value Estimation in Multi-Agent Reinforcement Learning [72.28520951105207]
  Overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning. We propose a novel regularization-based update scheme that penalizes large joint action-values deviating from a baseline. We show that our method provides a consistent performance improvement on a set of challenging StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2021-03-22T14:18:39Z)
• What Makes for Good Views for Contrastive Learning? [90.49736973404046]
  We argue that we should reduce the mutual information (MI) between views while keeping task-relevant information intact. We devise unsupervised and semi-supervised frameworks that learn effective views by aiming to reduce their MI. As a by-product, we achieve a new state-of-the-art accuracy on unsupervised pre-training for ImageNet classification.
  arXiv  Detail & Related papers  (2020-05-20T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.