Constructing Taxonomies from Pretrained Language Models

• URL: http://arxiv.org/abs/2010.12813v2
• Date: Sun, 18 Apr 2021 02:37:29 GMT
• Title: Constructing Taxonomies from Pretrained Language Models
• Authors: Catherine Chen, Kevin Lin, Dan Klein
• Abstract summary: We present a method for constructing taxonomic trees (e.g., WordNet) using pretrained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that reconciles those predictions into trees. We train our model on subtrees sampled from WordNet, and test on non-overlapping WordNet subtrees.
• Score: 52.53846972667636
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a method for constructing taxonomic trees (e.g., WordNet) using pretrained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that reconciles those predictions into trees. The parenthood prediction module produces likelihood scores for each potential parent-child pair, creating a graph of parent-child relation scores. The tree reconciliation module treats the task as a graph optimization problem and outputs the maximum spanning tree of this graph. We train our model on subtrees sampled from WordNet, and test on non-overlapping WordNet subtrees. We show that incorporating web-retrieved glosses can further improve performance. On the task of constructing subtrees of English WordNet, the model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best published result on this task. In addition, we convert the original English dataset into nine other languages using Open Multilingual WordNet and extend our results across these languages.

Related papers

• Unsupervised and Few-shot Parsing from Pretrained Language Models [56.33247845224995]
  We propose an Unsupervised constituent Parsing model that calculates an Out Association score solely based on the self-attention weight matrix learned in a pretrained language model. We extend the unsupervised models to few-shot parsing models that use a few annotated trees to learn better linear projection matrices for parsing. Our few-shot parsing model FPIO trained with only 20 annotated trees outperforms a previous few-shot parsing method trained with 50 annotated trees.
  arXiv  Detail & Related papers  (2022-06-10T10:29:15Z)
• LyS_ACoru\~na at SemEval-2022 Task 10: Repurposing Off-the-Shelf Tools for Sentiment Analysis as Semantic Dependency Parsing [10.355938901584567]
  This paper addresses the problem of structured sentiment analysis using a bi-affine semantic dependency. For the monolingual setup, we considered: (i) training on a single treebank, and (ii) relaxing the setup by training on treebanks coming from different languages. For the zero-shot setup and a given target treebank, we relied on: (i) a word-level translation of available treebanks in other languages to get noisy, unlikely-grammatical, but annotated data. In the post-evaluation phase, we also trained cross-lingual models that simply merged all the English tree
  arXiv  Detail & Related papers  (2022-04-27T10:21:28Z)
• Multilingual Syntax-aware Language Modeling through Dependency Tree Conversion [12.758523394180695]
  We study the effect on neural language models (LMs) performance across nine conversion methods and five languages. On average, the performance of our best model represents a 19 % increase in accuracy over the worst choice across all languages. Our experiments highlight the importance of choosing the right tree formalism, and provide insights into making an informed decision.
  arXiv  Detail & Related papers  (2022-04-19T03:56:28Z)
• TGIF: Tree-Graph Integrated-Format Parser for Enhanced UD with Two-Stage Generic- to Individual-Language Finetuning [18.71574180551552]
  We present our contribution to the IWPT 2021 shared task on parsing into enhanced Universal Dependencies. Our main system component is a hybrid tree-graph that integrates predictions of spanning trees for the enhanced graphs with additional graph edges not present in the spanning trees.
  arXiv  Detail & Related papers  (2021-07-14T18:00:08Z)
• Second-Order Unsupervised Neural Dependency Parsing [52.331561380948564]
  Most unsupervised dependencys are based on first-order probabilistic generative models that only consider local parent-child information. Inspired by second-order supervised dependency parsing, we proposed a second-order extension of unsupervised neural dependency models that incorporate grandparent-child or sibling information. Our joint model achieves a 10% improvement over the previous state-of-the-art on the full WSJ test set.
  arXiv  Detail & Related papers  (2020-10-28T03:01:33Z)
• Recursive Top-Down Production for Sentence Generation with Latent Trees [77.56794870399288]
  We model the production property of context-free grammars for natural and synthetic languages. We present a dynamic programming algorithm that marginalises over latent binary tree structures with $N$ leaves. We also present experimental results on German-English translation on the Multi30k dataset.
  arXiv  Detail & Related papers  (2020-10-09T17:47:16Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)
• Exploiting Syntactic Structure for Better Language Modeling: A Syntactic Distance Approach [78.77265671634454]
  We make use of a multi-task objective, i.e., the models simultaneously predict words as well as ground truth parse trees in a form called "syntactic distances" Experimental results on the Penn Treebank and Chinese Treebank datasets show that when ground truth parse trees are provided as additional training signals, the model is able to achieve lower perplexity and induce trees with better quality.
  arXiv  Detail & Related papers  (2020-05-12T15:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.