Simple Contrastive Representation Adversarial Learning for NLP Tasks
- URL: http://arxiv.org/abs/2111.13301v1
- Date: Fri, 26 Nov 2021 03:16:09 GMT
- Title: Simple Contrastive Representation Adversarial Learning for NLP Tasks
- Authors: Deshui Miao and Jiaqi Zhang and Wenbo Xie and Jian Song and Xin Li and
Lijuan Jia and Ning Guo
- Abstract summary: Two novel frameworks, supervised contrastive adversarial learning (SCAL) and unsupervised SCAL (USCAL), are proposed.
We employ it to Transformer-based models for natural language understanding, sentence semantic textual similarity and adversarial learning tasks.
Experimental results on GLUE benchmark tasks show that our fine-tuned supervised method outperforms BERT$_base$ over 1.75%.
- Score: 17.12062566060011
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Self-supervised learning approach like contrastive learning is attached great
attention in natural language processing. It uses pairs of training data
augmentations to build a classification task for an encoder with well
representation ability. However, the construction of learning pairs over
contrastive learning is much harder in NLP tasks. Previous works generate
word-level changes to form pairs, but small transforms may cause notable
changes on the meaning of sentences as the discrete and sparse nature of
natural language. In this paper, adversarial training is performed to generate
challenging and harder learning adversarial examples over the embedding space
of NLP as learning pairs. Using contrastive learning improves the
generalization ability of adversarial training because contrastive loss can
uniform the sample distribution. And at the same time, adversarial training
also enhances the robustness of contrastive learning. Two novel frameworks,
supervised contrastive adversarial learning (SCAL) and unsupervised SCAL
(USCAL), are proposed, which yields learning pairs by utilizing the adversarial
training for contrastive learning. The label-based loss of supervised tasks is
exploited to generate adversarial examples while unsupervised tasks bring
contrastive loss. To validate the effectiveness of the proposed framework, we
employ it to Transformer-based models for natural language understanding,
sentence semantic textual similarity and adversarial learning tasks.
Experimental results on GLUE benchmark tasks show that our fine-tuned
supervised method outperforms BERT$_{base}$ over 1.75\%. We also evaluate our
unsupervised method on semantic textual similarity (STS) tasks, and our method
gets 77.29\% with BERT$_{base}$. The robustness of our approach conducts
state-of-the-art results under multiple adversarial datasets on NLI tasks.
Related papers
- DenoSent: A Denoising Objective for Self-Supervised Sentence
Representation Learning [59.4644086610381]
We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective.
By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form.
Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
arXiv Detail & Related papers (2024-01-24T17:48:45Z) - SCAT: Robust Self-supervised Contrastive Learning via Adversarial
Training for Text Classification [15.932462099791307]
We propose a novel learning framework called SCAT (Self-supervised Contrastive Learning via Adversarial Training)
SCAT modifies random augmentations of the data in a fully labelfree manner to generate adversarial examples.
Our results show that SCAT can not only train robust language models from scratch, but it can also significantly improve the robustness of existing pre-trained language models.
arXiv Detail & Related papers (2023-07-04T05:41:31Z) - Improving Contrastive Learning of Sentence Embeddings from AI Feedback [43.56070504980024]
Supervised contrastive learning can produce more accurate sample pairs with human feedback labels.
Our method utilizes AI feedback from large pre-trained language models to construct sample pairs with fine-grained sample similarity scores.
Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity tasks.
arXiv Detail & Related papers (2023-05-03T06:26:13Z) - Multi-Task Self-Supervised Time-Series Representation Learning [3.31490164885582]
Time-series representation learning can extract representations from data with temporal dynamics and sparse labels.
We propose a new time-series representation learning method by combining the advantages of self-supervised tasks.
We evaluate the proposed framework on three downstream tasks: time-series classification, forecasting, and anomaly detection.
arXiv Detail & Related papers (2023-03-02T07:44:06Z) - PointACL:Adversarial Contrastive Learning for Robust Point Clouds
Representation under Adversarial Attack [73.3371797787823]
Adversarial contrastive learning (ACL) is considered an effective way to improve the robustness of pre-trained models.
We present our robust aware loss function to train self-supervised contrastive learning framework adversarially.
We validate our method, PointACL on downstream tasks, including 3D classification and 3D segmentation with multiple datasets.
arXiv Detail & Related papers (2022-09-14T22:58:31Z) - Contrastive Instruction-Trajectory Learning for Vision-Language
Navigation [66.16980504844233]
A vision-language navigation (VLN) task requires an agent to reach a target with the guidance of natural language instruction.
Previous works fail to discriminate the similarities and discrepancies across instruction-trajectory pairs and ignore the temporal continuity of sub-instructions.
We propose a Contrastive Instruction-Trajectory Learning framework that explores invariance across similar data samples and variance across different ones to learn distinctive representations for robust navigation.
arXiv Detail & Related papers (2021-12-08T06:32:52Z) - Contrastive Learning for Fair Representations [50.95604482330149]
Trained classification models can unintentionally lead to biased representations and predictions.
Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise.
We propose a method for mitigating bias by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations.
arXiv Detail & Related papers (2021-09-22T10:47:51Z) - Adversarial Training with Contrastive Learning in NLP [0.0]
We propose adversarial training with contrastive learning (ATCL) to adversarially train a language processing task.
The core idea is to make linear perturbations in the embedding space of the input via fast gradient methods (FGM) and train the model to keep the original and perturbed representations close via contrastive learning.
The results show not only an improvement in the quantitative (perplexity and BLEU) scores when compared to the baselines, but ATCL also achieves good qualitative results in the semantic level for both tasks.
arXiv Detail & Related papers (2021-09-19T07:23:45Z) - Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
We introduce a novel incremental false negative detection for self-supervised contrastive learning.
During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives.
Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
arXiv Detail & Related papers (2021-06-07T15:29:14Z) - Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness.
We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
arXiv Detail & Related papers (2020-10-26T04:44:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.