CLEAR: Contrastive Learning for Sentence Representation

• URL: http://arxiv.org/abs/2012.15466v1
• Date: Thu, 31 Dec 2020 06:40:13 GMT
• Title: CLEAR: Contrastive Learning for Sentence Representation
• Authors: Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, Hao Ma
• Abstract summary: We propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies. These augmentations include word and span deletion, reordering, and substitution. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.
• Score: 41.867438597420346
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Related papers

• DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning [59.4644086610381]
  We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
  arXiv  Detail & Related papers  (2024-01-24T17:48:45Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• Alleviating Over-smoothing for Unsupervised Sentence Representation [96.19497378628594]
  We present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting.
  arXiv  Detail & Related papers  (2023-05-09T11:00:02Z)
• Sentence Representation Learning with Generative Objective rather than Contrastive Objective [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
  arXiv  Detail & Related papers  (2022-10-16T07:47:46Z)
• SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning [14.028140579482688]
  SimCSE surprisingly dominates discrete augmentations such as cropping, word deletion, and synonym replacement as reported. We develop three simple yet effective discrete sentence augmentation schemes: punctuation insertion, modal verbs, and double negation. Results support the superiority of the proposed methods consistently.
  arXiv  Detail & Related papers  (2022-10-08T08:07:47Z)
• Probing for Understanding of English Verb Classes and Alternations in Large Pre-trained Language Models [4.243426191555036]
  We investigate the extent to which verb alternation classes are encoded in the embeddings of Large Pre-trained Language Models. We find that contextual embeddings from PLMs achieve astonishingly high accuracies on tasks across most classes.
  arXiv  Detail & Related papers  (2022-09-11T08:04:40Z)
• SLM: Learning a Discourse Language Representation with Sentence Unshuffling [53.42814722621715]
  We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation. We show that this feature of our model improves the performance of the original BERT by large margins.
  arXiv  Detail & Related papers  (2020-10-30T13:33:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.