Instilling Type Knowledge in Language Models via Multi-Task QA

• URL: http://arxiv.org/abs/2204.13796v1
• Date: Thu, 28 Apr 2022 22:06:32 GMT
• Title: Instilling Type Knowledge in Language Models via Multi-Task QA
• Authors: Shuyang Li, Mukund Sridhar, Chandana Satya Prakash, Jin Cao, Wael Hamza, Julian McAuley
• Abstract summary: We introduce a method to instill fine-grained type knowledge in language models with text-to-text pre-training on type-centric questions. We create the WikiWiki dataset: entities and passages from 10M Wikipedia articles linked to the Wikidata knowledge graph with 41K types. Models trained on WikiWiki achieve state-of-the-art performance in zero-shot dialog state tracking benchmarks, accurately infer entity types in Wikipedia articles, and can discover new types deemed useful by human judges.
• Score: 13.244420493711981
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding human language often necessitates understanding entities and their place in a taxonomy of knowledge -- their types. Previous methods to learn entity types rely on training classifiers on datasets with coarse, noisy, and incomplete labels. We introduce a method to instill fine-grained type knowledge in language models with text-to-text pre-training on type-centric questions leveraging knowledge base documents and knowledge graphs. We create the WikiWiki dataset: entities and passages from 10M Wikipedia articles linked to the Wikidata knowledge graph with 41K types. Models trained on WikiWiki achieve state-of-the-art performance in zero-shot dialog state tracking benchmarks, accurately infer entity types in Wikipedia articles, and can discover new types deemed useful by human judges.

Related papers

• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• Does Wikidata Support Analogical Reasoning? [17.68704739786042]
  We investigate whether the knowledge in Wikidata supports analogical reasoning. We show that Wikidata can be used to create data for analogy classification. We devise a set of metrics to guide an automatic method for extracting analogies from Wikidata.
  arXiv  Detail & Related papers  (2022-10-02T20:46:52Z)
• WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs [66.88232442007062]
  We introduce WikiDes, a dataset to generate short descriptions of Wikipedia articles. The dataset consists of over 80k English samples on 6987 topics. Our paper shows a practical impact on Wikipedia and Wikidata since there are thousands of missing descriptions.
  arXiv  Detail & Related papers  (2022-09-27T01:28:02Z)
• SKILL: Structured Knowledge Infusion for Large Language Models [46.34209061364282]
  We propose a method to infuse structured knowledge into large language models (LLMs) We show that models pre-trained on Wikidata KG with our method outperform the T5 baselines on FreebaseQA and WikiHop. We saw 3x improvement of exact match score on MetaQA task compared to T5 baseline.
  arXiv  Detail & Related papers  (2022-05-17T09:12:22Z)
• Assessing the quality of sources in Wikidata across languages: a hybrid approach [64.05097584373979]
  We run a series of microtasks experiments to evaluate a large corpus of references, sampled from Wikidata triples with labels in several languages. We use a consolidated, curated version of the crowdsourced assessments to train several machine learning models to scale up the analysis to the whole of Wikidata. The findings help us ascertain the quality of references in Wikidata, and identify common challenges in defining and capturing the quality of user-generated multilingual structured data on the web.
  arXiv  Detail & Related papers  (2021-09-20T10:06:46Z)
• Mining Knowledge for Natural Language Inference from Wikipedia Categories [53.26072815839198]
  We introduce WikiNLI: a resource for improving model performance on NLI and LE tasks. It contains 428,899 pairs of phrases constructed from naturally annotated category hierarchies in Wikipedia. We show that we can improve strong baselines such as BERT and RoBERTa by pretraining them on WikiNLI and transferring the models on downstream tasks.
  arXiv  Detail & Related papers  (2020-10-03T00:45:01Z)
• Crosslingual Topic Modeling with WikiPDA [15.198979978589476]
  We present Wikipedia-based Polyglot Dirichlet Allocation (WikiPDA) It learns to represent Wikipedia articles written in any language as distributions over a common set of language-independent topics. We show its utility in two applications: a study of topical biases in 28 Wikipedia editions, and crosslingual supervised classification.
  arXiv  Detail & Related papers  (2020-09-23T15:19:27Z)
• ALICE: Active Learning with Contrastive Natural Language Explanations [69.03658685761538]
  We propose Active Learning with Contrastive Explanations (ALICE) to improve data efficiency in learning. ALICE learns to first use active learning to select the most informative pairs of label classes to elicit contrastive natural language explanations. It extracts knowledge from these explanations using a semantically extracted knowledge.
  arXiv  Detail & Related papers  (2020-09-22T01:02:07Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)
• Learning Cross-Context Entity Representations from Text [9.981223356176496]
  We investigate the use of a fill-in-the-blank task to learn context independent representations of entities from text contexts. We show that large scale training of neural models allows us to learn high quality entity representations. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions.
  arXiv  Detail & Related papers  (2020-01-11T15:30:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.