Multi-Modal Knowledge Graph Construction and Application: A Survey

• URL: http://arxiv.org/abs/2202.05786v1
• Date: Fri, 11 Feb 2022 17:31:12 GMT
• Title: Multi-Modal Knowledge Graph Construction and Application: A Survey
• Authors: Xiangru Zhu, Zhixu Li, Xiaodan Wang, Xueyao Jiang, Penglei Sun, Xuwu Wang, Yanghua Xiao, Nicholas Jing Yuan
• Abstract summary: Multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. We first give definitions of MMKGs constructed by texts and images, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strength and weakness of different solutions.
• Score: 17.203534055251435
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strength and weakness of different solutions. We finalize this survey with open research problems relevant to MMKGs.

Related papers

• Explore then Determine: A GNN-LLM Synergy Framework for Reasoning over Knowledge Graph [38.31983923708175]
  This paper focuses on the Question Answering over Knowledge Graph (KGQA) task. It proposes an Explore-then-Determine (EtD) framework that synergizes Large Language Models with graph neural networks (GNNs) for reasoning over KGs. EtD achieves state-of-the-art performance and generates faithful reasoning results.
  arXiv  Detail & Related papers  (2024-06-03T09:38:28Z)
• Empowering Small-Scale Knowledge Graphs: A Strategy of Leveraging General-Purpose Knowledge Graphs for Enriched Embeddings [3.7759315989669058]
  We introduce a framework for enriching embeddings of small-scale domain-specific Knowledge Graphs with well-established general-purpose KGs. Experimental evaluations demonstrate a notable enhancement, with up to a 44% increase observed in the Hits@10 metric. This relatively unexplored research direction can catalyze more frequent incorporation of KGs in knowledge-intensive tasks.
  arXiv  Detail & Related papers  (2024-05-17T12:46:23Z)
• MyGO: Discrete Modality Information as Fine-Grained Tokens for Multi-modal Knowledge Graph Completion [51.80447197290866]
  We introduce MyGO to process, fuse, and augment the fine-grained modality information from MMKGs. MyGO tokenizes multi-modal raw data as fine-grained discrete tokens and learns entity representations with a cross-modal entity encoder. Experiments on standard MMKGC benchmarks reveal that our method surpasses 20 of the latest models.
  arXiv  Detail & Related papers  (2024-04-15T05:40:41Z)
• Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [61.8716670402084]
  This survey focuses on KG-aware research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, and Multi-Modal Knowledge Graph (MM4KG) Our review includes two primary task categories: KG-aware multi-modal learning tasks, and intrinsic MMKG tasks. For most of these tasks, we provide definitions, evaluation benchmarks, and additionally outline essential insights for conducting relevant research.
  arXiv  Detail & Related papers  (2024-02-08T04:04:36Z)
• On the Evolution of Knowledge Graphs: A Survey and Perspective [11.061075842989817]
  Knowledge graphs (KGs) are structured representations of diversified knowledge. They are widely used in various intelligent applications. We provide a comprehensive survey on the evolution of various types of KGs and techniques for knowledge extraction and reasoning. We propose our perspective on the future directions of knowledge engineering.
  arXiv  Detail & Related papers  (2023-10-07T14:46:51Z)
• A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal [57.8455911689554]
  Knowledge graph reasoning (KGR) aims to deduce new facts from existing facts based on mined logic rules underlying knowledge graphs (KGs) It has been proven to significantly benefit the usage of KGs in many AI applications, such as question answering, recommendation systems, and etc.
  arXiv  Detail & Related papers  (2022-12-12T08:40:04Z)
• A Survey on Visual Transfer Learning using Knowledge Graphs [0.8701566919381223]
  This survey focuses on visual transfer learning approaches using knowledge graphs (KGs) KGs can represent auxiliary knowledge either in an underlying graph-structured schema or in a vector-based knowledge graph embedding. We provide a broad overview of knowledge graph embedding methods and describe several joint training objectives suitable to combine them with high dimensional visual embeddings.
  arXiv  Detail & Related papers  (2022-01-27T20:19:55Z)
• Dynamic Semantic Graph Construction and Reasoning for Explainable Multi-hop Science Question Answering [50.546622625151926]
  We propose a new framework to exploit more valid facts while obtaining explainability for multi-hop QA. Our framework contains three new ideas: (a) tt AMR-SG, an AMR-based Semantic Graph, constructed by candidate fact AMRs to uncover any hop relations among question, answer and multiple facts, (b) a novel path-based fact analytics approach exploiting tt AMR-SG to extract active facts from a large fact pool to answer questions, and (c) a fact-level relation modeling leveraging graph convolution network (GCN) to guide the reasoning process.
  arXiv  Detail & Related papers  (2021-05-25T09:14:55Z)
• KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion [99.47414073164656]
  A comprehensive knowledge graph (KG) contains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to "simulate" human's abilities on knowledge abstraction, concretization, and completion. We propose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty.
  arXiv  Detail & Related papers  (2020-04-28T16:21:57Z)
• A Survey on Knowledge Graphs: Representation, Acquisition and Applications [89.78089494738002]
  We review research topics about 1) knowledge graph representation learning, 2) knowledge acquisition and completion, 3) temporal knowledge graph, and 4) knowledge-aware applications. For knowledge acquisition, especially knowledge graph completion, embedding methods, path inference, and logical rule reasoning, are reviewed. We explore several emerging topics, including meta learning, commonsense reasoning, and temporal knowledge graphs.
  arXiv  Detail & Related papers  (2020-02-02T13:17:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.