Pairwise Supervised Contrastive Learning of Sentence Representations
- URL: http://arxiv.org/abs/2109.05424v1
- Date: Sun, 12 Sep 2021 04:12:16 GMT
- Title: Pairwise Supervised Contrastive Learning of Sentence Representations
- Authors: Dejiao Zhang, Shang-Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati,
Andrew O. Arnold, Bing Xiang
- Abstract summary: PairSupCon aims to bridge semantic entailment and contradiction understanding with high-level categorical concept encoding.
We evaluate it on various downstream tasks that involve understanding sentence semantics at different granularities.
- Score: 20.822509446824125
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Many recent successes in sentence representation learning have been achieved
by simply fine-tuning on the Natural Language Inference (NLI) datasets with
triplet loss or siamese loss. Nevertheless, they share a common weakness:
sentences in a contradiction pair are not necessarily from different semantic
categories. Therefore, optimizing the semantic entailment and contradiction
reasoning objective alone is inadequate to capture the high-level semantic
structure. The drawback is compounded by the fact that the vanilla siamese or
triplet losses only learn from individual sentence pairs or triplets, which
often suffer from bad local optima. In this paper, we propose PairSupCon, an
instance discrimination based approach aiming to bridge semantic entailment and
contradiction understanding with high-level categorical concept encoding. We
evaluate PairSupCon on various downstream tasks that involve understanding
sentence semantics at different granularities. We outperform the previous
state-of-the-art method with $10\%$--$13\%$ averaged improvement on eight
clustering tasks, and $5\%$--$6\%$ averaged improvement on seven semantic
textual similarity (STS) tasks.
Related papers
- GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding [66.5538429726564]
Self-supervised 3D representation learning aims to learn effective representations from large-scale unlabeled point clouds.
We propose GroupContrast, a novel approach that combines segment grouping and semantic-aware contrastive learning.
arXiv Detail & Related papers (2024-03-14T17:59:59Z) - DenoSent: A Denoising Objective for Self-Supervised Sentence
Representation Learning [59.4644086610381]
We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective.
By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form.
Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
arXiv Detail & Related papers (2024-01-24T17:48:45Z) - Identical and Fraternal Twins: Fine-Grained Semantic Contrastive
Learning of Sentence Representations [6.265789210037749]
We introduce a novel Identical and Fraternal Twins of Contrastive Learning framework, capable of simultaneously adapting to various positive pairs generated by different augmentation techniques.
We also present proof-of-concept experiments combined with the contrastive objective to prove the validity of the proposed Twins Loss.
arXiv Detail & Related papers (2023-07-20T15:02:42Z) - RankCSE: Unsupervised Sentence Representations Learning via Learning to
Rank [54.854714257687334]
We propose a novel approach, RankCSE, for unsupervised sentence representation learning.
It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework.
An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
arXiv Detail & Related papers (2023-05-26T08:27:07Z) - Relational Sentence Embedding for Flexible Semantic Matching [86.21393054423355]
We present Sentence Embedding (RSE), a new paradigm to discover further the potential of sentence embeddings.
RSE is effective and flexible in modeling sentence relations and outperforms a series of state-of-the-art embedding methods.
arXiv Detail & Related papers (2022-12-17T05:25:17Z) - Sentence Representation Learning with Generative Objective rather than
Contrastive Objective [86.01683892956144]
We propose a novel generative self-supervised learning objective based on phrase reconstruction.
Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
arXiv Detail & Related papers (2022-10-16T07:47:46Z) - Keywords and Instances: A Hierarchical Contrastive Learning Framework
Unifying Hybrid Granularities for Text Generation [59.01297461453444]
We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text.
Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
arXiv Detail & Related papers (2022-05-26T13:26:03Z) - Pair-Level Supervised Contrastive Learning for Natural Language
Inference [11.858875850000457]
We propose a Pair-level Supervised Contrastive Learning approach (PairSCL)
A contrastive learning objective is designed to distinguish the varied classes of sentence pairs by pulling those in one class together and pushing apart the pairs in other classes.
We evaluate PairSCL on two public datasets of NLI where the accuracy of PairSCL outperforms other methods by 2.1% on average.
arXiv Detail & Related papers (2022-01-26T13:34:52Z) - SimCSE: Simple Contrastive Learning of Sentence Embeddings [10.33373737281907]
This paper presents SimCSE, a contrastive learning framework for embeddings.
We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective.
We then incorporate annotated pairs from NLI datasets into contrastive learning by using "entailment" pairs as positives and "contradiction" pairs as hard negatives.
arXiv Detail & Related papers (2021-04-18T11:27:08Z) - Reevaluating Adversarial Examples in Natural Language [20.14869834829091]
We analyze the outputs of two state-of-the-art synonym substitution attacks.
We find that their perturbations often do not preserve semantics, and 38% introduce grammatical errors.
With constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points.
arXiv Detail & Related papers (2020-04-25T03:09:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.