PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings

• URL: http://arxiv.org/abs/2201.12093v1
• Date: Fri, 28 Jan 2022 13:02:41 GMT
• Title: PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings
• Authors: Qiyu Wu, Chongyang Tao, Tao Shen, Can Xu, Xiubo Geng, Daxin Jiang
• Abstract summary: We propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL can perform peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability.
• Score: 69.87899694963251
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning sentence embeddings in an unsupervised manner is fundamental in natural language processing. Recent common practice is to couple pre-trained language models with unsupervised contrastive learning, whose success relies on augmenting a sentence with a semantically-close positive instance to construct contrastive pairs. Nonetheless, existing approaches usually depend on a mono-augmenting strategy, which causes learning shortcuts towards the augmenting biases and thus corrupts the quality of sentence embeddings. A straightforward solution is resorting to more diverse positives from a multi-augmenting strategy, while an open question remains about how to unsupervisedly learn from the diverse positives but with uneven augmenting qualities in the text field. As one answer, we propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL can perform peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability and an effective way to learn from diverse augmentations. Experiments on STS benchmarks verify the effectiveness of our PCL against its competitors in unsupervised sentence embeddings.

Related papers

• Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding [17.12010497289781]
  We argue that leveraging multiple positives should be considered for multi-lingual sentence embeddings. We propose a novel approach, named MPCL, to effectively utilize multiple positive instances to improve the learning of multi-lingual sentence embeddings.
  arXiv  Detail & Related papers  (2023-09-16T08:54:30Z)
• DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective [1.351603931922027]
  We argue that effectively eliminating the influence of various biases is crucial for learning high-quality sentence embeddings. We propose a novel contrastive framework for sentence embedding, termed DebCSE, which can eliminate the impact of these biases.
  arXiv  Detail & Related papers  (2023-09-14T02:43:34Z)
• Alleviating Over-smoothing for Unsupervised Sentence Representation [96.19497378628594]
  We present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue. Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting.
  arXiv  Detail & Related papers  (2023-05-09T11:00:02Z)
• Improving Contrastive Learning of Sentence Embeddings from AI Feedback [43.56070504980024]
  Supervised contrastive learning can produce more accurate sample pairs with human feedback labels. Our method utilizes AI feedback from large pre-trained language models to construct sample pairs with fine-grained sample similarity scores. Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity tasks.
  arXiv  Detail & Related papers  (2023-05-03T06:26:13Z)
• Adversarial Contrastive Learning via Asymmetric InfoNCE [64.42740292752069]
  We propose to treat adversarial samples unequally when contrasted with an asymmetric InfoNCE objective. In the asymmetric fashion, the adverse impacts of conflicting objectives between CL and adversarial learning can be effectively mitigated. Experiments show that our approach consistently outperforms existing Adversarial CL methods.
  arXiv  Detail & Related papers  (2022-07-18T04:14:36Z)
• Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation [59.01297461453444]
  We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
  arXiv  Detail & Related papers  (2022-05-26T13:26:03Z)
• A Simple Contrastive Learning Objective for Alleviating Neural Text Degeneration [56.64703901898937]
  We propose a new contrastive token learning objective that inherits the advantages of cross-entropy and unlikelihood training. Comprehensive experiments on language modeling and open-domain dialogue generation tasks show that the proposed contrastive token objective yields less repetitive texts.
  arXiv  Detail & Related papers  (2022-05-05T08:50:50Z)
• Simple Contrastive Representation Adversarial Learning for NLP Tasks [17.12062566060011]
  Two novel frameworks, supervised contrastive adversarial learning (SCAL) and unsupervised SCAL (USCAL), are proposed. We employ it to Transformer-based models for natural language understanding, sentence semantic textual similarity and adversarial learning tasks. Experimental results on GLUE benchmark tasks show that our fine-tuned supervised method outperforms BERT$_base$ over 1.75%.
  arXiv  Detail & Related papers  (2021-11-26T03:16:09Z)
• Contrastive Learning with Adversarial Examples [79.39156814887133]
  Contrastive learning (CL) is a popular technique for self-supervised learning (SSL) of visual representations. This paper introduces a new family of adversarial examples for constrastive learning and using these examples to define a new adversarial training algorithm for SSL, denoted as CLAE.
  arXiv  Detail & Related papers  (2020-10-22T20:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.