MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning

• URL: http://arxiv.org/abs/2209.01416v1
• Date: Sat, 3 Sep 2022 13:07:02 GMT
• Title: MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning
• Authors: Shangfei Zheng, Weiqing Wang, Jianfeng Qu, Hongzhi Yin, Wei Chen and Lei Zhao
• Abstract summary: We propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning) The model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; and (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process. The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.
• Score: 40.60328470622483
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal knowledge graphs (MKGs) include not only the relation triplets, but also related multi-modal auxiliary data (i.e., texts and images), which enhance the diversity of knowledge. However, the natural incompleteness has significantly hindered the applications of MKGs. To tackle the problem, existing studies employ the embedding-based reasoning models to infer the missing knowledge after fusing the multi-modal features. However, the reasoning performance of these methods is limited due to the following problems: (1) ineffective fusion of multi-modal auxiliary features; (2) lack of complex reasoning ability as well as inability to conduct the multi-hop reasoning which is able to infer more missing knowledge. To overcome these problems, we propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning). Specifically, the model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process, based on the features obtained in component (1). The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.

Related papers

• Multimodal Reasoning with Multimodal Knowledge Graph [19.899398342533722]
  Multimodal reasoning with large language models (LLMs) often suffers from hallucinations and the presence of deficient or outdated knowledge. We propose the Multimodal Reasoning with Multimodal Knowledge Graph (MR-MKG) method to learn rich and semantic knowledge across modalities.
  arXiv  Detail & Related papers  (2024-06-04T07:13:23Z)
• Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning [51.80447197290866]
  Learning high-quality multi-modal entity representations is an important goal of multi-modal knowledge graph (MMKG) representation learning. Existing methods focus on crafting elegant entity-wise multi-modal fusion strategies. We introduce a novel framework with Mixture of Modality Knowledge experts (MoMoK) to learn adaptive multi-modal entity representations.
  arXiv  Detail & Related papers  (2024-05-27T06:36:17Z)
• Cantor: Inspiring Multimodal Chain-of-Thought of MLLM [83.6663322930814]
  We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks. We propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture. Our experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance.
  arXiv  Detail & Related papers  (2024-04-24T17:59:48Z)
• Zero-Shot Relational Learning for Multimodal Knowledge Graphs [31.215889061734295]
  One of the major challenges is inference on newly discovered relations without associated training data. Existing works fail to support the leverage of multimodal information and leave the problem unexplored. We propose a novel end-to-end framework, consisting of three components, i.e., multimodal learner, structure consolidator embedding generator, to integrate diverse multimodal information and knowledge graph structures.
  arXiv  Detail & Related papers  (2024-04-09T11:14:45Z)
• MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models [61.479419734006825]
  We introduce MAGDi, a new method for structured distillation of the reasoning interactions between multiple Large Language Model (LLM) agents into smaller LMs. Experiments on seven widely used commonsense and math reasoning benchmarks show that MAGDi improves the reasoning capabilities of smaller models. We conduct extensive analyses to show that MAGDi enhances the generalizability to out-of-domain tasks, scales positively with the size and strength of the base student model, and obtains larger improvements when applying self-consistency.
  arXiv  Detail & Related papers  (2024-02-02T18:35:14Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Multimodal Analogical Reasoning over Knowledge Graphs [43.76819868795101]
  We introduce the new task of multimodal analogical reasoning over knowledge graphs. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory.
  arXiv  Detail & Related papers  (2022-10-01T16:24:15Z)
• Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering [35.40919477319811]
  We propose a novel knowledge-aware approach that equips pre-trained language models with a multi-hop relational reasoning module. It performs multi-hop, multi-relational reasoning over subgraphs extracted from external knowledge graphs. It unifies path-based reasoning methods and graph neural networks to achieve better interpretability and scalability.
  arXiv  Detail & Related papers  (2020-05-01T23:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.