Endowing Language Models with Multimodal Knowledge Graph Representations
- URL: http://arxiv.org/abs/2206.13163v1
- Date: Mon, 27 Jun 2022 10:10:42 GMT
- Title: Endowing Language Models with Multimodal Knowledge Graph Representations
- Authors: Ningyuan Huang and Yash R. Deshpande and Yibo Liu and Houda Alberts
and Kyunghyun Cho and Clara Vania and Iacer Calixto
- Abstract summary: We use the recently released VisualSem KG as our external knowledge repository.
We retrieve entities from the KG and use their multimodal representations to improve downstream task performance.
- Score: 47.22480859519051
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a method to make natural language understanding models more
parameter efficient by storing knowledge in an external knowledge graph (KG)
and retrieving from this KG using a dense index. Given (possibly multilingual)
downstream task data, e.g., sentences in German, we retrieve entities from the
KG and use their multimodal representations to improve downstream task
performance. We use the recently released VisualSem KG as our external
knowledge repository, which covers a subset of Wikipedia and WordNet entities,
and compare a mix of tuple-based and graph-based algorithms to learn entity and
relation representations that are grounded on the KG multimodal information. We
demonstrate the usefulness of the learned entity representations on two
downstream tasks, and show improved performance on the multilingual named
entity recognition task by $0.3\%$--$0.7\%$ F1, while we achieve up to $2.5\%$
improvement in accuracy on the visual sense disambiguation task. All our code
and data are available in: \url{https://github.com/iacercalixto/visualsem-kg}.
Related papers
- Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
We model the subgraph retrieval task as a conditional generation task handled by small language models.
Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models.
Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
arXiv Detail & Related papers (2024-10-08T15:22:36Z) - Retrieval-Augmented Language Model for Extreme Multi-Label Knowledge Graph Link Prediction [2.6749568255705656]
Extrapolation in large language models (LLMs) for open-ended inquiry encounters two pivotal issues.
Existing works attempt to tackle the problem by augmenting the input of a smaller language model with information from a knowledge graph.
We propose a new task, the extreme multi-label KG link prediction task, to enable a model to perform extrapolation with multiple responses.
arXiv Detail & Related papers (2024-05-21T10:10:56Z) - BanglaAutoKG: Automatic Bangla Knowledge Graph Construction with Semantic Neural Graph Filtering [6.05977559550463]
Knowledge Graphs (KGs) have proven essential in information processing and reasoning applications.
Despite being widely used globally, Bangla is relatively underrepresented in KGs due to a lack of comprehensive datasets.
We propose BanglaAutoKG, a pioneering framework that is able to automatically construct Bengali KGs from any Bangla text.
arXiv Detail & Related papers (2024-04-04T15:31:21Z) - FedMKGC: Privacy-Preserving Federated Multilingual Knowledge Graph
Completion [21.4302940596294]
Knowledge graph completion (KGC) aims to predict missing facts in knowledge graphs (KGs)
Previous methods that rely on transferring raw data among KGs raise privacy concerns.
We propose a new federated learning framework that implicitly aggregates knowledge from multiple KGs without demanding raw data exchange and entity alignment.
arXiv Detail & Related papers (2023-12-17T08:09:27Z) - DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System
for Multilingual Named Entity Recognition [94.90258603217008]
The MultiCoNER RNum2 shared task aims to tackle multilingual named entity recognition (NER) in fine-grained and noisy scenarios.
Previous top systems in the MultiCoNER RNum1 either incorporate the knowledge bases or gazetteers.
We propose a unified retrieval-augmented system (U-RaNER) for fine-grained multilingual NER.
arXiv Detail & Related papers (2023-05-05T16:59:26Z) - Deep Bidirectional Language-Knowledge Graph Pretraining [159.9645181522436]
DRAGON is a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale.
Our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities.
arXiv Detail & Related papers (2022-10-17T18:02:52Z) - Few-shot Knowledge Graph-to-Text Generation with Pretrained Language
Models [42.38563175680914]
This paper studies how to automatically generate a natural language text that describes the facts in knowledge graph (KG)
Considering the few-shot setting, we leverage the excellent capacities of pretrained language models (PLMs) in language understanding and generation.
arXiv Detail & Related papers (2021-06-03T06:48:00Z) - Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced
Language Model Pre-training [22.534866015730664]
We verbalize the entire English Wikidata KG.
We show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora.
arXiv Detail & Related papers (2020-10-23T22:14:50Z) - VisualSem: A High-quality Knowledge Graph for Vision and Language [48.47370435793127]
We release VisualSem: a high-quality knowledge graph (KG)
VisualSem includes nodes with multilingual glosses, multiple illustrative images, and visually relevant relations.
We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG.
arXiv Detail & Related papers (2020-08-20T18:20:29Z) - Exploiting Structured Knowledge in Text via Graph-Guided Representation
Learning [73.0598186896953]
We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs.
Building upon entity-level masked language models, our first contribution is an entity masking scheme.
In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
arXiv Detail & Related papers (2020-04-29T14:22:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.