DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

• URL: http://arxiv.org/abs/2204.10298v1
• Date: Thu, 21 Apr 2022 17:32:01 GMT
• Title: DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
• Authors: Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Solja\v{c}i\'c, Shang-Wen Li, Wen-tau Yih, Yoon Kim, James Glass
• Abstract summary: DiffCSE is an unsupervised contrastive learning framework for learning sentence embeddings. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods.
• Score: 51.274478128525686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other "harmful" types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE by 2.3 absolute points on semantic textual similarity tasks.

Related papers

• DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning [59.4644086610381]
  We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
  arXiv  Detail & Related papers  (2024-01-24T17:48:45Z)
• Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE [13.494159547236425]
  This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives.
  arXiv  Detail & Related papers  (2023-10-10T18:15:24Z)
• DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in the Debiasing Perspective [1.351603931922027]
  We argue that effectively eliminating the influence of various biases is crucial for learning high-quality sentence embeddings. We propose a novel contrastive framework for sentence embedding, termed DebCSE, which can eliminate the impact of these biases.
  arXiv  Detail & Related papers  (2023-09-14T02:43:34Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• Contrastive Learning of Sentence Embeddings from Scratch [26.002876719243464]
  We present SynCSE, a contrastive learning framework that trains sentence embeddings with synthesized data. Specifically, we explore utilizing large language models to synthesize the required data samples for contrastive learning. Experimental results on sentence similarity and reranking tasks indicate that both SynCSE-partial and SynCSE-scratch greatly outperform unsupervised baselines.
  arXiv  Detail & Related papers  (2023-05-24T11:56:21Z)
• InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings [61.77760317554826]
  This paper proposes an information-d contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large.
  arXiv  Detail & Related papers  (2022-10-08T15:53:19Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning may yield powerful enough sentence representation and achieve performance in Sentence Textual Similarity tasks on par with contrastive learning.
  arXiv  Detail & Related papers  (2022-04-20T10:00:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.