UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining

• URL: http://arxiv.org/abs/2202.13469v1
• Date: Sun, 27 Feb 2022 22:43:06 GMT
• Title: UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Mining
• Authors: Jiacheng Li, Jingbo Shang, Julian McAuley
• Abstract summary: UCTopic is a novel unsupervised contrastive learning framework for context-aware phrase representations and topic mining. It is pretrained in a large scale to distinguish if the contexts of two phrase mentions have the same semantics. It outperforms the state-of-the-art phrase representation model by 38.2% NMI in average on four entity cluster-ing tasks.
• Score: 27.808028645942827
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-quality phrase representations are essential to finding topics and related terms in documents (a.k.a. topic mining). Existing phrase representation learning methods either simply combine unigram representations in a context-free manner or rely on extensive annotations to learn context-aware knowledge. In this paper, we propose UCTopic, a novel unsupervised contrastive learning framework for context-aware phrase representations and topic mining. UCTopic is pretrained in a large scale to distinguish if the contexts of two phrase mentions have the same semantics. The key to pretraining is positive pair construction from our phrase-oriented assumptions. However, we find traditional in-batch negatives cause performance decay when finetuning on a dataset with small topic numbers. Hence, we propose cluster-assisted contrastive learning(CCL) which largely reduces noisy negatives by selecting negatives from clusters and further improves phrase representations for topics accordingly. UCTopic outperforms the state-of-the-art phrase representation model by 38.2% NMI in average on four entity cluster-ing tasks. Comprehensive evaluation on topic mining shows that UCTopic can extract coherent and diverse topical phrases.

Related papers

• GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding [66.5538429726564]
  Self-supervised 3D representation learning aims to learn effective representations from large-scale unlabeled point clouds. We propose GroupContrast, a novel approach that combines segment grouping and semantic-aware contrastive learning.
  arXiv  Detail & Related papers  (2024-03-14T17:59:59Z)
• Vocabulary-Defined Semantics: Latent Space Clustering for Improving In-Context Learning [32.178931149612644]
  In-context learning enables language models to adapt to downstream data or incorporate tasks by few samples as demonstrations within the prompts. However, the performance of in-context learning can be unstable depending on the quality, format, or order of demonstrations. We propose a novel approach "vocabulary-defined semantics"
  arXiv  Detail & Related papers  (2024-01-29T14:29:48Z)
• DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning [59.4644086610381]
  We propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks.
  arXiv  Detail & Related papers  (2024-01-24T17:48:45Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• Clustering-Aware Negative Sampling for Unsupervised Sentence Representation [24.15096466098421]
  ClusterNS is a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified K-means clustering algorithm to supply hard negatives and recognize in-batch false negatives during training.
  arXiv  Detail & Related papers  (2023-05-17T02:06:47Z)
• Semantic-aware Contrastive Learning for More Accurate Semantic Parsing [32.74456368167872]
  We propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations. Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines.
  arXiv  Detail & Related papers  (2023-01-19T07:04:32Z)
• Sentence Representation Learning with Generative Objective rather than Contrastive Objective [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
  arXiv  Detail & Related papers  (2022-10-16T07:47:46Z)
• MGIMN: Multi-Grained Interactive Matching Network for Few-shot Text Classification [9.9875634964736]
  Text classification struggles to generalize to unseen classes with very few labeled text instances per class. We propose a meta-learning based method MGIMN which performs instance-wise comparison followed by aggregation to generate class-wise matching vectors.
  arXiv  Detail & Related papers  (2022-04-11T08:58:55Z)
• Self-Supervised Learning Disentangled Group Representation as Feature [82.07737719232972]
  We show that existing Self-Supervised Learning (SSL) only disentangles simple augmentation features such as rotation and colorization. We propose an iterative SSL algorithm: Iterative Partition-based Invariant Risk Minimization (IP-IRM) We prove that IP-IRM converges to a fully disentangled representation and show its effectiveness on various benchmarks.
  arXiv  Detail & Related papers  (2021-10-28T16:12:33Z)
• On the Interplay Between Fine-tuning and Composition in Transformers [7.513100214864645]
  We investigate the impact of fine-tuning on the capacity of contextualized embeddings to capture phrase meaning information. Specifically, we fine-tune models on an adversarial paraphrase classification task with high lexical overlap, and on a sentiment classification task. We find that fine-tuning largely fails to benefit compositionality in these representations, though training on sentiment yields a small, localized benefit for certain models.
  arXiv  Detail & Related papers  (2021-05-31T01:49:56Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.