Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completion

• URL: http://arxiv.org/abs/2407.02867v1
• Date: Wed, 3 Jul 2024 07:31:33 GMT
• Title: Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completion
• Authors: Yu Zhao, Ying Zhang, Baohang Zhou, Xinying Qian, Kehui Song, Xiangrui Cai,
• Abstract summary: A large number of studies have emerged for Multimodal Knowledge Graph Completion (MKGC) to predict the missing links in MKGs. Existing inductive approaches focus on learning textual entity representations, which neglect rich semantic information in visual modality. We propose a semantic neighbor retrieval-enhanced IMKGC framework, where the contrast brings the helpful semantic neighbors close.
• Score: 12.504140924121872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A large number of studies have emerged for Multimodal Knowledge Graph Completion (MKGC) to predict the missing links in MKGs. However, fewer studies have been proposed to study the inductive MKGC (IMKGC) involving emerging entities unseen during training. Existing inductive approaches focus on learning textual entity representations, which neglect rich semantic information in visual modality. Moreover, they focus on aggregating structural neighbors from existing KGs, which of emerging entities are usually limited. However, the semantic neighbors are decoupled from the topology linkage and usually imply the true target entity. In this paper, we propose the IMKGC task and a semantic neighbor retrieval-enhanced IMKGC framework CMR, where the contrast brings the helpful semantic neighbors close, and then the memorize supports semantic neighbor retrieval to enhance inference. Specifically, we first propose a unified cross-modal contrastive learning to simultaneously capture the textual-visual and textual-textual correlations of query-entity pairs in a unified representation space. The contrastive learning increases the similarity of positive query-entity pairs, therefore making the representations of helpful semantic neighbors close. Then, we explicitly memorize the knowledge representations to support the semantic neighbor retrieval. At test time, we retrieve the nearest semantic neighbors and interpolate them to the query-entity similarity distribution to augment the final prediction. Extensive experiments validate the effectiveness of CMR on three inductive MKGC datasets. Codes are available at https://github.com/OreOZhao/CMR.

Related papers

• Non-Neighbors Also Matter to Kriging: A New Contrastive-Prototypical Learning [24.701170582359104]
  Existing works assume that neighbors' information offers the basis for estimating the attributes of the unobserved target. We propose Contrastive-Prototypical'' self-supervised learning for Kriging to refine valuable information from neighbors and recycle the one from non-neighbors.
  arXiv  Detail & Related papers  (2024-01-23T11:46:31Z)
• Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos [63.94040814459116]
  Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence. We propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps. We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations.
  arXiv  Detail & Related papers  (2023-08-19T09:12:13Z)
• Knowledge-Enhanced Hierarchical Information Correlation Learning for Multi-Modal Rumor Detection [82.94413676131545]
  We propose a novel knowledge-enhanced hierarchical information correlation learning approach (KhiCL) for multi-modal rumor detection. KhiCL exploits cross-modal joint dictionary to transfer the heterogeneous unimodality features into the common feature space. It extracts visual and textual entities from images and text, and designs a knowledge relevance reasoning strategy.
  arXiv  Detail & Related papers  (2023-06-28T06:08:20Z)
• Learnable Pillar-based Re-ranking for Image-Text Retrieval [119.9979224297237]
  Image-text retrieval aims to bridge the modality gap and retrieve cross-modal content based on semantic similarities. Re-ranking, a popular post-processing practice, has revealed the superiority of capturing neighbor relations in single-modality retrieval tasks. We propose a novel learnable pillar-based re-ranking paradigm for image-text retrieval.
  arXiv  Detail & Related papers  (2023-04-25T04:33:27Z)
• FECANet: Boosting Few-Shot Semantic Segmentation with Feature-Enhanced Context-Aware Network [48.912196729711624]
  Few-shot semantic segmentation is the task of learning to locate each pixel of a novel class in a query image with only a few annotated support images. We propose a Feature-Enhanced Context-Aware Network (FECANet) to suppress the matching noise caused by inter-class local similarity. In addition, we propose a novel correlation reconstruction module that encodes extra correspondence relations between foreground and background and multi-scale context semantic features.
  arXiv  Detail & Related papers  (2023-01-19T16:31:13Z)
• Relational Message Passing for Fully Inductive Knowledge Graph Completion [37.29833710603933]
  In knowledge graph completion (KGC), predicting triples involving emerging entities and/or relations, which are unseen when KG embeddings are learned, has become a critical challenge. Subgraph reasoning with message passing is a promising and popular solution. We propose a new method named RMPI which uses a novel Message Passing network for fully available KGC.
  arXiv  Detail & Related papers  (2022-10-08T10:35:52Z)
• Learning Attention-based Representations from Multiple Patterns for Relation Prediction in Knowledge Graphs [2.4028383570062606]
  AEMP is a novel model for learning contextualized representations by acquiring entities' context information. AEMP either outperforms or competes with state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-07T10:53:35Z)
• Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings [29.94706167233985]
  kNN-KGE is a new knowledge graph embedding approach with pre-trained language models. We compute the nearest neighbors based on the distance in the entity embedding space from the knowledge store.
  arXiv  Detail & Related papers  (2022-01-14T17:35:16Z)
• Learning the Implicit Semantic Representation on Graph-Structured Data [57.670106959061634]
  Existing representation learning methods in graph convolutional networks are mainly designed by describing the neighborhood of each node as a perceptual whole. We propose a Semantic Graph Convolutional Networks (SGCN) that explores the implicit semantics by learning latent semantic-paths in graphs.
  arXiv  Detail & Related papers  (2021-01-16T16:18:43Z)
• Neighborhood Matching Network for Entity Alignment [71.24217694278616]
  Neighborhood Matching Network (NMN) is a novel entity alignment framework. NMN estimates the similarities between entities to capture both the topological structure and the neighborhood difference. It first uses a novel graph sampling method to distill a discriminative neighborhood for each entity. It then adopts a cross-graph neighborhood matching module to jointly encode the neighborhood difference for a given entity pair.
  arXiv  Detail & Related papers  (2020-05-12T08:26:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.