Multimodal Analogical Reasoning over Knowledge Graphs

• URL: http://arxiv.org/abs/2210.00312v1
• Date: Sat, 1 Oct 2022 16:24:15 GMT
• Title: Multimodal Analogical Reasoning over Knowledge Graphs
• Authors: Ningyu Zhang, Lei Li, Xiang Chen, Xiaozhuan Liang, Shumin Deng, Huajun Chen
• Abstract summary: We introduce the new task of multimodal analogical reasoning over knowledge graphs. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory.
• Score: 43.76819868795101
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance.

Related papers

• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation Learning [51.80447197290866]
  Learning high-quality multi-modal entity representations is an important goal of multi-modal knowledge graph (MMKG) representation learning. Existing methods focus on crafting elegant entity-wise multi-modal fusion strategies. We introduce a novel framework with Mixture of Modality Knowledge experts (MoMoK) to learn adaptive multi-modal entity representations.
  arXiv  Detail & Related papers  (2024-05-27T06:36:17Z)
• Cantor: Inspiring Multimodal Chain-of-Thought of MLLM [83.6663322930814]
  We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks. We propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture. Our experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance.
  arXiv  Detail & Related papers  (2024-04-24T17:59:48Z)
• Zero-Shot Relational Learning for Multimodal Knowledge Graphs [31.215889061734295]
  One of the major challenges is inference on newly discovered relations without associated training data. Existing works fail to support the leverage of multimodal information and leave the problem unexplored. We propose a novel end-to-end framework, consisting of three components, i.e., multimodal learner, structure consolidator embedding generator, to integrate diverse multimodal information and knowledge graph structures.
  arXiv  Detail & Related papers  (2024-04-09T11:14:45Z)
• Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction [8.169359626365619]
  We generate a textitchain of thought (CoT) -- a sequence of intermediate reasoning steps. We present a novel conditional prompt distillation method to assimilate the commonsense reasoning ability from large language models. Our approach attains state-of-the-art accuracy and manifests a plethora of advantages concerning interpretability, data efficiency, and cross-domain generalization.
  arXiv  Detail & Related papers  (2023-06-25T04:33:56Z)
• Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions [68.6358773622615]
  This paper provides an overview of the computational and theoretical foundations of multimodal machine learning. We propose a taxonomy of 6 core technical challenges: representation, alignment, reasoning, generation, transference, and quantification. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches.
  arXiv  Detail & Related papers  (2022-09-07T19:21:19Z)
• MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning [40.60328470622483]
  We propose a novel model entitled MMKGR (Multi-hop Multi-modal Knowledge Graph Reasoning) The model contains the following two components: (1) a unified gate-attention network which is designed to generate effective multi-modal complementary features through sufficient attention interaction and noise reduction; and (2) a complementary feature-aware reinforcement learning method which is proposed to predict missing elements by performing the multi-hop reasoning process. The experimental results demonstrate that MMKGR outperforms the state-of-the-art approaches in the MKG reasoning task.
  arXiv  Detail & Related papers  (2022-09-03T13:07:02Z)
• Multimodal foundation models are better simulators of the human brain [65.10501322822881]
  We present a newly-designed multimodal foundation model pre-trained on 15 million image-text pairs. We find that both visual and lingual encoders trained multimodally are more brain-like compared with unimodal ones.
  arXiv  Detail & Related papers  (2022-08-17T12:36:26Z)
• Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering [35.40919477319811]
  We propose a novel knowledge-aware approach that equips pre-trained language models with a multi-hop relational reasoning module. It performs multi-hop, multi-relational reasoning over subgraphs extracted from external knowledge graphs. It unifies path-based reasoning methods and graph neural networks to achieve better interpretability and scalability.
  arXiv  Detail & Related papers  (2020-05-01T23:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.