Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding

• URL: http://arxiv.org/abs/2211.03348v1
• Date: Mon, 7 Nov 2022 07:50:30 GMT
• Title: Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding
• Authors: Jiali Zeng, Yongjing Yin, Yufan Jiang, Shuangzhi Wu, Yunbo Cao
• Abstract summary: We propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP) We construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of prompts. Using a contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes.
• Score: 24.350264932134078
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning has become a new paradigm for unsupervised sentence embeddings. Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In this paper, we propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP). Specifically, with the help of prompts, we construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of the prompts. Using a prototypical contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes, and far apart from the negative prototypes as well as the prototypes of other sentences. Extensive experimental results on semantic textual similarity, transfer, and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines. Code is available at https://github.com/lemon0830/promptCSE.

Related papers

• Advancing Interpretability in Text Classification through Prototype Learning [1.9526476410335776]
  ProtoLens is a prototype-based model that provides fine-grained, sub-sentence level interpretability for text classification. ProtoLens uses a Prototype-aware Span Extraction module to identify relevant text spans. ProtoLens provides interpretable predictions while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2024-10-23T03:53:46Z)
• Negative Prototypes Guided Contrastive Learning for WSOD [8.102080369924911]
  Weakly Supervised Object Detection (WSOD) with only image-level annotation has recently attracted wide attention. We propose the Negative Prototypes Guided Contrastive learning architecture. Our proposed method achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-06-04T08:16:26Z)
• CAPro: Webly Supervised Learning with Cross-Modality Aligned Prototypes [93.71909293023663]
  Cross-modality Aligned Prototypes (CAPro) is a unified contrastive learning framework to learn visual representations with correct semantics. CAPro achieves new state-of-the-art performance and exhibits robustness to open-set recognition.
  arXiv  Detail & Related papers  (2023-10-15T07:20:22Z)
• ConsPrompt: Exploiting Contrastive Samples for Fewshot Prompt Learning [37.219617741198334]
  We explore utilizing suitable contrastive samples and multi-degree contrastive learning methods to improve the robustness of the prompt representation. Our results exhibit state-of-the-art performance in different few-shot settings, and the ablation experiments also certify the effectiveness of utilizing multi-degree contrastive learning in the prompt-based fine-tuning process.
  arXiv  Detail & Related papers  (2022-11-08T09:29:45Z)
• Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation [59.01297461453444]
  We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
  arXiv  Detail & Related papers  (2022-05-26T13:26:03Z)
• Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning may yield powerful enough sentence representation and achieve performance in Sentence Textual Similarity tasks on par with contrastive learning.
  arXiv  Detail & Related papers  (2022-04-20T10:00:46Z)
• Robust Textual Embedding against Word-level Adversarial Attacks [15.235449552083043]
  We propose a novel robust training method, termed Fast Triplet Metric Learning (FTML) We show that FTML can significantly promote the model robustness against various advanced adversarial attacks. Our work shows the great potential of improving the textual robustness through robust word embedding.
  arXiv  Detail & Related papers  (2022-02-28T14:25:00Z)
• Dual Prototypical Contrastive Learning for Few-shot Semantic Segmentation [55.339405417090084]
  We propose a dual prototypical contrastive learning approach tailored to the few-shot semantic segmentation (FSS) task. The main idea is to encourage the prototypes more discriminative by increasing inter-class distance while reducing intra-class distance in prototype feature space. We demonstrate that the proposed dual contrastive learning approach outperforms state-of-the-art FSS methods on PASCAL-5i and COCO-20i datasets.
  arXiv  Detail & Related papers  (2021-11-09T08:14:50Z)
• Prototypical Representation Learning for Relation Extraction [56.501332067073065]
  This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data. We learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art relational models.
  arXiv  Detail & Related papers  (2021-03-22T08:11:43Z)
• Prototypical Contrastive Learning of Unsupervised Representations [171.3046900127166]
  Prototypical Contrastive Learning (PCL) is an unsupervised representation learning method. PCL implicitly encodes semantic structures of the data into the learned embedding space. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T09:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.