Guiding Symbolic Natural Language Grammar Induction via Transformer-Based Sequence Probabilities

• URL: http://arxiv.org/abs/2005.12533v1
• Date: Tue, 26 May 2020 06:18:47 GMT
• Title: Guiding Symbolic Natural Language Grammar Induction via Transformer-Based Sequence Probabilities
• Authors: Ben Goertzel, Andres Suarez Madrigal, Gino Yu
• Abstract summary: A novel approach to automated learning of syntactic rules governing natural languages is proposed. This method exploits the learned linguistic knowledge in transformers, without any reference to their inner representations. We show a proof-of-concept example of our proposed technique, using it to guide unsupervised symbolic link-grammar induction methods.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A novel approach to automated learning of syntactic rules governing natural languages is proposed, based on using probabilities assigned to sentences (and potentially longer word sequences) by transformer neural network language models to guide symbolic learning processes like clustering and rule induction. This method exploits the learned linguistic knowledge in transformers, without any reference to their inner representations; hence, the technique is readily adaptable to the continuous appearance of more powerful language models. We show a proof-of-concept example of our proposed technique, using it to guide unsupervised symbolic link-grammar induction methods drawn from our prior research.

Related papers

• A distributional simplicity bias in the learning dynamics of transformers [50.91742043564049]
  We show that transformers, trained on natural language data, also display a simplicity bias. Specifically, they sequentially learn many-body interactions among input tokens, reaching a saturation point in the prediction error for low-degree interactions. This approach opens up the possibilities of studying how interactions of different orders in the data affect learning, in natural language processing and beyond.
  arXiv  Detail & Related papers  (2024-10-25T15:39:34Z)
• Leveraging Grammar Induction for Language Understanding and Generation [7.459693992079273]
  We introduce an unsupervised grammar induction method for language understanding and generation. We construct a grammar to induce constituency structures and dependency relations, which is simultaneously trained on downstream tasks. We evaluate and apply our method to multiple machine translation tasks natural language understanding tasks.
  arXiv  Detail & Related papers  (2024-10-07T09:57:59Z)
• Faithful and Plausible Natural Language Explanations for Image Classification: A Pipeline Approach [10.54430941755474]
  This paper proposes a post-hoc natural language explanation method that can be applied to any CNN-based classification system. By analysing influential neurons and the corresponding activation maps, the method generates a faithful description of the classifier's decision process. Experimental results show that the NLEs constructed by our method are significantly more plausible and faithful.
  arXiv  Detail & Related papers  (2024-07-30T15:17:15Z)
• On Conditional and Compositional Language Model Differentiable Prompting [75.76546041094436]
  Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks. We propose a new model, Prompt Production System (PRopS), which learns to transform task instructions or input metadata, into continuous prompts.
  arXiv  Detail & Related papers  (2023-07-04T02:47:42Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• Neuro-Symbolic Hierarchical Rule Induction [12.610497441047395]
  We propose an efficient interpretable neuro-symbolic model to solve Inductive Logic Programming (ILP) problems. In this model, which is built from a set of meta-rules organised in a hierarchical structure, first-order rules are invented by learning embeddings to match facts and body predicates of a meta-rule. We empirically validate our model on various tasks (ILP, visual genome, reinforcement learning) against several state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-12-26T17:02:14Z)
• Skill Induction and Planning with Latent Language [94.55783888325165]
  We formulate a generative model of action sequences in which goals generate sequences of high-level subtask descriptions. We describe how to train this model using primarily unannotated demonstrations by parsing demonstrations into sequences of named high-level subtasks. In trained models, the space of natural language commands indexes a library of skills; agents can use these skills to plan by generating high-level instruction sequences tailored to novel goals.
  arXiv  Detail & Related papers  (2021-10-04T15:36:32Z)
• SLM: Learning a Discourse Language Representation with Sentence Unshuffling [53.42814722621715]
  We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation. We show that this feature of our model improves the performance of the original BERT by large margins.
  arXiv  Detail & Related papers  (2020-10-30T13:33:41Z)
• Generative latent neural models for automatic word alignment [0.0]
  Variational autoencoders have been recently used in various of natural language processing to learn in an unsupervised way latent representations that are useful for language generation tasks. In this paper, we study these models for the task of word alignment and propose and assess several evolutions of a vanilla variational autoencoders. We demonstrate that these techniques can yield competitive results as compared to Giza++ and to a strong neural network alignment system for two language pairs.
  arXiv  Detail & Related papers  (2020-09-28T07:54:09Z)
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation [2.578242050187029]
  We propose a new paradigm for introducing a syntactic inductive bias into neural text generation. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity.
  arXiv  Detail & Related papers  (2020-04-05T14:29:40Z)
• Learning Compositional Rules via Neural Program Synthesis [67.62112086708859]
  We present a neuro-symbolic model which learns entire rule systems from a small set of examples. Instead of directly predicting outputs from inputs, we train our model to induce the explicit system of rules governing a set of previously seen examples.
  arXiv  Detail & Related papers  (2020-03-12T01:06:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.