UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training

• URL: http://arxiv.org/abs/2302.06891v4
• Date: Sat, 28 Sep 2024 15:32:42 GMT
• Title: UKnow: A Unified Knowledge Protocol with Multimodal Knowledge Graph Datasets for Reasoning and Vision-Language Pre-Training
• Abstract summary: This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data. We categorize data knowledge into five unit types, namely, in-image, in-text, cross-image, cross-text, and image-text. Following UKnow protocol, we collect, from public international news, a large-scale multimodal knowledge graph dataset.
• Score: 48.184673241046134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work presents a unified knowledge protocol, called UKnow, which facilitates knowledge-based studies from the perspective of data. Particularly focusing on visual and linguistic modalities, we categorize data knowledge into five unit types, namely, in-image, in-text, cross-image, cross-text, and image-text, and set up an efficient pipeline to help construct the multimodal knowledge graph from any data collection. Thanks to the logical information naturally contained in knowledge graph, organizing datasets under UKnow format opens up more possibilities of data usage compared to the commonly used image-text pairs. Following UKnow protocol, we collect, from public international news, a large-scale multimodal knowledge graph dataset that consists of 1,388,568 nodes (with 571,791 vision-related ones) and 3,673,817 triplets. The dataset is also annotated with rich event tags, including 11 coarse labels and 9,185 fine labels. Experiments on 4 benchmarks demonstrate the potential of UKnow in supporting common-sense reasoning and boosting vision-language pre-training with a single dataset, benefiting from its unified form of knowledge organization. See Appendix to download the dataset.

Related papers

• Knowledge Graphs: The Future of Data Integration and Insightful Discovery [0.0]
  Knowledge graphs organize data by linking points, enabling researchers to combine diverse information sources into a single database. They capture complex relationships inherent in unstructured data sources, offering a semantic framework for diverse entities. Strategies for developing knowledge graphs include using seed data, named entity recognition, and relationship extraction.
  arXiv  Detail & Related papers  (2024-12-17T19:49:33Z)
• Bridging Local Details and Global Context in Text-Attributed Graphs [62.522550655068336]
  GraphBridge is a framework that bridges local and global perspectives by leveraging contextual textual information. Our method achieves state-of-theart performance, while our graph-aware token reduction module significantly enhances efficiency and solves scalability issues.
  arXiv  Detail & Related papers  (2024-06-18T13:35:25Z)
• Multilingual Diversity Improves Vision-Language Representations [66.41030381363244]
  Pre-training on this dataset outperforms using English-only or English-dominated datasets on ImageNet. On a geographically diverse task like GeoDE, we also observe improvements across all regions, with the biggest gain coming from Africa.
  arXiv  Detail & Related papers  (2024-05-27T08:08:51Z)
• VisionKG: Unleashing the Power of Visual Datasets via Knowledge Graph [2.3143591448419074]
  Vision Knowledge Graph (VisionKG) is a novel resource that interlinks, organizes and manages visual datasets via knowledge graphs and Semantic Web technologies. VisionKG currently contains 519 million RDF triples that describe approximately 40 million entities.
  arXiv  Detail & Related papers  (2023-09-24T11:19:13Z)
• End-to-End Learning on Multimodal Knowledge Graphs [0.0]
  We propose a multimodal message passing network which learns end-to-end from the structure of graphs. Our model uses dedicated (neural) encoders to naturally learn embeddings for node features belonging to five different types of modalities. Our results indicate that end-to-end multimodal learning from any arbitrary knowledge graph is indeed possible.
  arXiv  Detail & Related papers  (2023-09-03T13:16:18Z)
• Learning Representations without Compositional Assumptions [79.12273403390311]
  We propose a data-driven approach that learns feature set dependencies by representing feature sets as graph nodes and their relationships as learnable edges. We also introduce LEGATO, a novel hierarchical graph autoencoder that learns a smaller, latent graph to aggregate information from multiple views dynamically.
  arXiv  Detail & Related papers  (2023-05-31T10:36:10Z)
• EventNarrative: A large-scale Event-centric Dataset for Knowledge Graph-to-Text Generation [8.216976747904726]
  EventNarrative consists of approximately 230,000 graphs and their corresponding natural language text, 6 times larger than the current largest parallel dataset. Our aim is two-fold: help break new ground in event-centric research where data is lacking, and to give researchers a well-defined, large-scale dataset.
  arXiv  Detail & Related papers  (2021-10-30T15:39:20Z)
• GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph [32.129005474301735]
  Contextual information is often the key to structure such real world data, and we propose to use it in form of a knowledge graph. We propose a novel use of a knowledge graph, that is constructed on annotated data and pseudo-labeled data. With label propagation, we boost artwork classification by training a model using a graph convolutional network.
  arXiv  Detail & Related papers  (2021-05-25T11:50:05Z)
• Knowledge-Guided Multi-Label Few-Shot Learning for General Image Recognition [75.44233392355711]
  KGGR framework exploits prior knowledge of statistical label correlations with deep neural networks. It first builds a structured knowledge graph to correlate different labels based on statistical label co-occurrence. Then, it introduces the label semantics to guide learning semantic-specific features. It exploits a graph propagation network to explore graph node interactions.
  arXiv  Detail & Related papers  (2020-09-20T15:05:29Z)
• ENT-DESC: Entity Description Generation by Exploring Knowledge Graph [53.03778194567752]
  In practice, the input knowledge could be more than enough, since the output description may only cover the most significant knowledge. We introduce a large-scale and challenging dataset to facilitate the study of such a practical scenario in KG-to-text. We propose a multi-graph structure that is able to represent the original graph information more comprehensively.
  arXiv  Detail & Related papers  (2020-04-30T14:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.