Something Old, Something New: Grammar-based CCG Parsing with Transformer Models

• URL: http://arxiv.org/abs/2109.10044v1
• Date: Tue, 21 Sep 2021 09:20:12 GMT
• Title: Something Old, Something New: Grammar-based CCG Parsing with Transformer Models
• Authors: Stephen Clark
• Abstract summary: This report describes the parsing problem for Combinatory Categorial Grammar (CCG) It shows how a combination of Transformer-based neural models and a symbolic CCG grammar can lead to substantial gains over existing approaches. It also documents a 20-year research program, showing how NLP methods have evolved over this time.
• Score: 13.035551917639692
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This report describes the parsing problem for Combinatory Categorial Grammar (CCG), showing how a combination of Transformer-based neural models and a symbolic CCG grammar can lead to substantial gains over existing approaches. The report also documents a 20-year research program, showing how NLP methods have evolved over this time. The staggering accuracy improvements provided by neural models for CCG parsing can be seen as a reflection of the improvements seen in NLP more generally. The report provides a minimal introduction to CCG and CCG parsing, with many pointers to the relevant literature. It then describes the CCG supertagging problem, and some recent work from Tian et al. (2020) which applies Transformer-based models to supertagging with great effect. I use this existing model to develop a CCG multitagger, which can serve as a front-end to an existing CCG parser. Simply using this new multitagger provides substantial gains in parsing accuracy. I then show how a Transformer-based model from the parsing literature can be combined with the grammar-based CCG parser, setting a new state-of-the-art for the CCGbank parsing task of almost 93% F-score for labelled dependencies, with complete sentence accuracies of over 50%.

Related papers

• Fast Context-Biasing for CTC and Transducer ASR models with CTC-based Word Spotter [57.64003871384959]
  This work presents a new approach to fast context-biasing with CTC-based Word Spotter. The proposed method matches CTC log-probabilities against a compact context graph to detect potential context-biasing candidates. The results demonstrate a significant acceleration of the context-biasing recognition with a simultaneous improvement in F-score and WER.
  arXiv  Detail & Related papers  (2024-06-11T09:37:52Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• Separate-and-Aggregate: A Transformer-based Patch Refinement Model for Knowledge Graph Completion [28.79628925695775]
  We propose a novel Transformer-based Patch Refinement Model (PatReFormer) for Knowledge Graph completion. We conduct experiments on four popular KGC benchmarks, WN18RR, FB15k-237, YAGO37 and DB100K. The experimental results show significant performance improvement from existing KGC methods on standard KGC evaluation metrics.
  arXiv  Detail & Related papers  (2023-07-11T06:27:13Z)
• From the One, Judge of the Whole: Typed Entailment Graph Construction with Predicate Generation [69.91691115264132]
  Entailment Graphs (EGs) are constructed to indicate context-independent entailment relations in natural languages. In this paper, we propose a multi-stage method, Typed Predicate-Entailment Graph Generator (TP-EGG) to tackle this problem. Experiments on benchmark datasets show that TP-EGG can generate high-quality and scale-controllable entailment graphs.
  arXiv  Detail & Related papers  (2023-06-07T05:46:19Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• Pre-training Transformers for Knowledge Graph Completion [81.4078733132239]
  We introduce a novel inductive KG representation model (iHT) for learning transferable representation for knowledge graphs. iHT consists of a entity encoder (e.g., BERT) and a neighbor-aware relational scoring function both parameterized by Transformers. Our approach achieves new state-of-the-art results on matched evaluations, with a relative improvement of more than 25% in mean reciprocal rank over previous SOTA models.
  arXiv  Detail & Related papers  (2023-03-28T02:10:37Z)
• CSynGEC: Incorporating Constituent-based Syntax for Grammatical Error Correction with a Tailored GEC-Oriented Parser [22.942594068051488]
  This work considers another mainstream syntax formalism, i.e. constituent-based syntax. We first propose an extended constituent-based syntax scheme to accommodate errors in ungrammatical sentences. Then, we automatically obtain constituency trees of ungrammatical sentences to train a GEC-oriented constituency.
  arXiv  Detail & Related papers  (2022-11-15T14:11:39Z)
• GN-Transformer: Fusing Sequence and Graph Representation for Improved Code Summarization [0.0]
  We propose a novel method, GN-Transformer, to learn end-to-end on a fused sequence and graph modality. The proposed methods achieve state-of-the-art performance in two code summarization datasets and across three automatic code summarization metrics.
  arXiv  Detail & Related papers  (2021-11-17T02:51:37Z)
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization [57.798070356553936]
  HETFORMER is a Transformer-based pre-trained model with multi-granularity sparse attentions for extractive summarization. Experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1.
  arXiv  Detail & Related papers  (2021-10-12T22:42:31Z)
• DisCoDisCo at the DISRPT2021 Shared Task: A System for Discourse Segmentation, Classification, and Connective Detection [4.371388370559826]
  Our system, called DisCoDisCo, enhances contextualized word embeddings with hand-crafted features. Results on relation classification suggest strong performance on the new 2021 benchmark. A partial evaluation of multiple pre-trained Transformer-based language models indicates that models pre-trained on the Next Sentence Prediction task are optimal for relation classification.
  arXiv  Detail & Related papers  (2021-09-20T18:11:05Z)
• Supertagging Combinatory Categorial Grammar with Attentive Graph Convolutional Networks [34.74687603029737]
  We propose attentive graph convolutional networks to enhance neural CCG supertagging through a novel solution of leveraging contextual information. Experiments performed on the CCGbank demonstrate that our approach outperforms all previous studies in terms of both supertagging and parsing.
  arXiv  Detail & Related papers  (2020-10-13T01:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.