Contextualized Representations Using Textual Encyclopedic Knowledge
- URL: http://arxiv.org/abs/2004.12006v2
- Date: Tue, 13 Jul 2021 05:39:18 GMT
- Title: Contextualized Representations Using Textual Encyclopedic Knowledge
- Authors: Mandar Joshi, Kenton Lee, Yi Luan, Kristina Toutanova
- Abstract summary: We show that integrating background knowledge from text is effective for tasks focusing on factual reasoning.
On TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable RoBERTa models.
- Score: 23.49437524363581
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a method to represent input texts by contextualizing them jointly
with dynamically retrieved textual encyclopedic background knowledge from
multiple documents. We apply our method to reading comprehension tasks by
encoding questions and passages together with background sentences about the
entities they mention. We show that integrating background knowledge from text
is effective for tasks focusing on factual reasoning and allows direct reuse of
powerful pretrained BERT-style encoders. Moreover, knowledge integration can be
further improved with suitable pretraining via a self-supervised masked
language model objective over words in background-augmented input text. On
TriviaQA, our approach obtains improvements of 1.6 to 3.1 F1 over comparable
RoBERTa models which do not integrate background knowledge dynamically. On
MRQA, a large collection of diverse QA datasets, we see consistent gains
in-domain along with large improvements out-of-domain on BioASQ (2.1 to 4.2
F1), TextbookQA (1.6 to 2.0 F1), and DuoRC (1.1 to 2.0 F1).
Related papers
- Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding [9.2433070542025]
Large language models (LLMs) tend to inadequately integrate input context during text generation.
We introduce a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples.
arXiv Detail & Related papers (2024-05-04T20:38:41Z) - KETM:A Knowledge-Enhanced Text Matching method [0.0]
We introduce a new model for text matching called the Knowledge Enhanced Text Matching model (KETM)
We use Wiktionary to retrieve the text word definitions as our external knowledge.
We fuse text and knowledge using a gating mechanism to learn the ratio of text and knowledge fusion.
arXiv Detail & Related papers (2023-08-11T17:08:14Z) - TegTok: Augmenting Text Generation via Task-specific and Open-world
Knowledge [83.55215993730326]
We propose augmenting TExt Generation via Task-specific and Open-world Knowledge (TegTok) in a unified framework.
Our model selects knowledge entries from two types of knowledge sources through dense retrieval and then injects them into the input encoding and output decoding stages respectively.
arXiv Detail & Related papers (2022-03-16T10:37:59Z) - Open Domain Question Answering over Virtual Documents: A Unified
Approach for Data and Text [62.489652395307914]
We use the data-to-text method as a means for encoding structured knowledge for knowledge-intensive applications, i.e. open-domain question answering (QA)
Specifically, we propose a verbalizer-retriever-reader framework for open-domain QA over data and text where verbalized tables from Wikipedia and triples from Wikidata are used as augmented knowledge sources.
We show that our Unified Data and Text QA, UDT-QA, can effectively benefit from the expanded knowledge index, leading to large gains over text-only baselines.
arXiv Detail & Related papers (2021-10-16T00:11:21Z) - KELM: Knowledge Enhanced Pre-Trained Language Representations with
Message Passing on Hierarchical Relational Graphs [26.557447199727758]
We propose a novel knowledge-aware language model framework based on fine-tuning process.
Our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT.
arXiv Detail & Related papers (2021-09-09T12:39:17Z) - Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation [49.89831914386982]
We propose a unified pre-trained language model (PLM) for all forms of text, including unstructured text, semi-structured text, and well-structured text.
Our approach outperforms the pre-training of plain text using only 1/4 of the data.
arXiv Detail & Related papers (2021-09-02T16:05:24Z) - Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced
Language Model Pre-training [22.534866015730664]
We verbalize the entire English Wikidata KG.
We show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora.
arXiv Detail & Related papers (2020-10-23T22:14:50Z) - Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
We present the first application of the BERTSum model to conversational language.
We generate abstractive summaries of narrated instructional videos across a wide variety of topics.
We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
arXiv Detail & Related papers (2020-08-21T20:59:34Z) - Exploiting Structured Knowledge in Text via Graph-Guided Representation
Learning [73.0598186896953]
We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs.
Building upon entity-level masked language models, our first contribution is an entity masking scheme.
In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
arXiv Detail & Related papers (2020-04-29T14:22:42Z) - Hierarchical Multi Task Learning with Subword Contextual Embeddings for
Languages with Rich Morphology [5.5217350574838875]
Morphological information is important for many sequence labeling tasks in Natural Language Processing (NLP)
We propose using subword contextual embeddings to capture morphological information for languages with rich morphology.
Our model outperforms previous state-of-the-art models on both tasks for the Turkish language.
arXiv Detail & Related papers (2020-04-25T22:55:56Z) - Exploring the Limits of Transfer Learning with a Unified Text-to-Text
Transformer [64.22926988297685]
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP)
In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
arXiv Detail & Related papers (2019-10-23T17:37:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.