Semantic-aware Contrastive Learning for More Accurate Semantic Parsing

• URL: http://arxiv.org/abs/2301.07919v1
• Date: Thu, 19 Jan 2023 07:04:32 GMT
• Title: Semantic-aware Contrastive Learning for More Accurate Semantic Parsing
• Authors: Shan Wu, Chunlei Xin, Bo Chen, Xianpei Han, Le Sun
• Abstract summary: We propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations. Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines.
• Score: 32.74456368167872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Since the meaning representations are detailed and accurate annotations which express fine-grained sequence-level semtantics, it is usually hard to train discriminative semantic parsers via Maximum Likelihood Estimation (MLE) in an autoregressive fashion. In this paper, we propose a semantic-aware contrastive learning algorithm, which can learn to distinguish fine-grained meaning representations and take the overall sequence-level semantic into consideration. Specifically, a multi-level online sampling algorithm is proposed to sample confusing and diverse instances. Three semantic-aware similarity functions are designed to accurately measure the distance between meaning representations as a whole. And a ranked contrastive loss is proposed to pull the representations of the semantic-identical instances together and push negative instances away. Experiments on two standard datasets show that our approach achieves significant improvements over MLE baselines and gets state-of-the-art performances by simply applying semantic-aware contrastive learning on a vanilla Seq2Seq model.

Related papers

• GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding [66.5538429726564]
  Self-supervised 3D representation learning aims to learn effective representations from large-scale unlabeled point clouds. We propose GroupContrast, a novel approach that combines segment grouping and semantic-aware contrastive learning.
  arXiv  Detail & Related papers  (2024-03-14T17:59:59Z)
• Semantic Contrastive Bootstrapping for Single-positive Multi-label Recognition [36.3636416735057]
  We present a semantic contrastive bootstrapping (Scob) approach to gradually recover the cross-object relationships. We then propose a recurrent semantic masked transformer to extract iconic object-level representations. Extensive experimental results demonstrate that the proposed joint learning framework surpasses the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-07-15T01:59:53Z)
• RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank [54.854714257687334]
  We propose a novel approach, RankCSE, for unsupervised sentence representation learning. It incorporates ranking consistency and ranking distillation with contrastive learning into a unified framework. An extensive set of experiments are conducted on both semantic textual similarity (STS) and transfer (TR) tasks.
  arXiv  Detail & Related papers  (2023-05-26T08:27:07Z)
• BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval [54.66399120084227]
  We propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Dense retrieval has shown promise in the first-stage retrieval process when trained on in-domain labeled datasets.
  arXiv  Detail & Related papers  (2023-05-18T15:43:09Z)
• Deep Semi-supervised Learning with Double-Contrast of Features and Semantics [2.2230089845369094]
  This paper proposes an end-to-end deep semi-supervised learning double contrast of semantic and feature. We leverage information theory to explain the rationality of double contrast of semantics and features.
  arXiv  Detail & Related papers  (2022-11-28T09:08:19Z)
• Self-Supervised Learning Disentangled Group Representation as Feature [82.07737719232972]
  We show that existing Self-Supervised Learning (SSL) only disentangles simple augmentation features such as rotation and colorization. We propose an iterative SSL algorithm: Iterative Partition-based Invariant Risk Minimization (IP-IRM) We prove that IP-IRM converges to a fully disentangled representation and show its effectiveness on various benchmarks.
  arXiv  Detail & Related papers  (2021-10-28T16:12:33Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)
• Disentangled Contrastive Learning for Learning Robust Textual Representations [13.880693856907037]
  We introduce the concept of momentum representation consistency to align features and leverage power normalization while conforming the uniformity. Our experimental results for the NLP benchmarks demonstrate that our approach can obtain better results compared with the baselines.
  arXiv  Detail & Related papers  (2021-04-11T03:32:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.