Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation

• URL: http://arxiv.org/abs/2205.13346v1
• Date: Thu, 26 May 2022 13:26:03 GMT
• Title: Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation
• Authors: Mingzhe Li, XieXiong Lin, Xiuying Chen, Jinxiong Chang, Qishen Zhang, Feng Wang, Taifeng Wang, Zhongyi Liu, Wei Chu, Dongyan Zhao, Rui Yan
• Abstract summary: We propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.
• Score: 59.01297461453444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning has achieved impressive success in generation tasks to militate the "exposure bias" problem and discriminatively exploit the different quality of references. Existing works mostly focus on contrastive learning on the instance-level without discriminating the contribution of each word, while keywords are the gist of the text and dominant the constrained mapping relationships. Hence, in this work, we propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Concretely, we first propose a keyword graph via contrastive correlations of positive-negative pairs to iteratively polish the keyword representations. Then, we construct intra-contrasts within instance-level and keyword-level, where we assume words are sampled nodes from a sentence distribution. Finally, to bridge the gap between independent contrast levels and tackle the common contrast vanishing problem, we propose an inter-contrast mechanism that measures the discrepancy between contrastive keyword nodes respectively to the instance distribution. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.

Related papers

• Saliency Attention and Semantic Similarity-Driven Adversarial Perturbation [0.0]
  Saliency Attention and Semantic Similarity driven adversarial Perturbation (SASSP) is designed to improve the effectiveness of contextual perturbations. Our proposed approach incorporates a three-pronged strategy for word selection and perturbation. SASSP has yielded a higher attack success rate and lower word perturbation rate.
  arXiv  Detail & Related papers  (2024-06-18T14:07:27Z)
• DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning [59.4644086610381]
  We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
  arXiv  Detail & Related papers  (2024-01-24T17:48:45Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents [61.63208012250885]
  We formulate recognizing semantic differences as a token-level regression task. We study three unsupervised approaches that rely on a masked language model. Our results show that an approach based on word alignment and sentence-level contrastive learning has a robust correlation to gold labels.
  arXiv  Detail & Related papers  (2023-05-22T17:58:04Z)
• ContrastNet: A Contrastive Learning Framework for Few-Shot Text Classification [40.808421462004866]
  We propose ContrastNet to tackle both discriminative representation and overfitting problems in few-shot text classification. Experiments on 8 few-shot text classification datasets show that ContrastNet outperforms the current state-of-the-art models.
  arXiv  Detail & Related papers  (2023-05-16T08:22:17Z)
• Dictionary-Assisted Supervised Contrastive Learning [0.0]
  We introduce the dictionary-assisted supervised contrastive learning (DASCL) objective, allowing researchers to leverage specialized dictionaries. The text is first keyword simplified: a common, fixed token replaces any word in the corpus that appears in the dictionary(ies) relevant to the concept of interest. DASCL and cross-entropy improves classification performance metrics in few-shot learning settings and social science applications.
  arXiv  Detail & Related papers  (2022-10-27T04:57:43Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• UniKeyphrase: A Unified Extraction and Generation Framework for Keyphrase Prediction [20.26899340581431]
  Keyphrase Prediction task aims at predicting several keyphrases that can summarize the main idea of the given document. Mainstream KP methods can be categorized into purely generative approaches and integrated models with extraction and generation. We propose UniKeyphrase, a novel end-to-end learning framework that jointly learns to extract and generate keyphrases.
  arXiv  Detail & Related papers  (2021-06-09T07:09:51Z)
• SLUA: A Super Lightweight Unsupervised Word Alignment Model via Cross-Lingual Contrastive Learning [79.91678610678885]
  We propose a super lightweight unsupervised word alignment model (SLUA) Experimental results on several public benchmarks demonstrate that our model achieves competitive, if not better, performance. Notably, we recognize our model as a pioneer attempt to unify bilingual word embedding and word alignments.
  arXiv  Detail & Related papers  (2021-02-08T05:54:11Z)
• Model Choices Influence Attributive Word Associations: A Semi-supervised Analysis of Static Word Embeddings [0.0]
  This work aims to assess attributive word associations across five different static word embedding architectures. Our results reveal that the choice of the context learning flavor during embedding training (CBOW vs skip-gram) impacts the word association distinguishability and word embeddings' sensitivity to deviations in the training corpora.
  arXiv  Detail & Related papers  (2020-12-14T22:27:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.