Unleashing the Power of Imbalanced Modality Information for Multi-modal
Knowledge Graph Completion
- URL: http://arxiv.org/abs/2402.15444v1
- Date: Thu, 22 Feb 2024 05:48:03 GMT
- Title: Unleashing the Power of Imbalanced Modality Information for Multi-modal
Knowledge Graph Completion
- Authors: Yichi Zhang, Zhuo Chen, Lei Liang, Huajun Chen, Wen Zhang
- Abstract summary: Multi-modal knowledge graph completion (MMKGC) aims to predict the missing triples in the multi-modal knowledge graphs.
We propose Adaptive Multi-modal Fusion and Modality Adversarial Training (AdaMF-MAT) to unleash the power of imbalanced modality information.
Our approach is a co-design of the MMKGC model and training strategy which can outperform 19 recent MMKGC methods.
- Score: 40.86196588992357
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Multi-modal knowledge graph completion (MMKGC) aims to predict the missing
triples in the multi-modal knowledge graphs by incorporating structural,
visual, and textual information of entities into the discriminant models. The
information from different modalities will work together to measure the triple
plausibility. Existing MMKGC methods overlook the imbalance problem of modality
information among entities, resulting in inadequate modal fusion and
inefficient utilization of the raw modality information. To address the
mentioned problems, we propose Adaptive Multi-modal Fusion and Modality
Adversarial Training (AdaMF-MAT) to unleash the power of imbalanced modality
information for MMKGC. AdaMF-MAT achieves multi-modal fusion with adaptive
modality weights and further generates adversarial samples by
modality-adversarial training to enhance the imbalanced modality information.
Our approach is a co-design of the MMKGC model and training strategy which can
outperform 19 recent MMKGC methods and achieve new state-of-the-art results on
three public MMKGC benchmarks. Our code and data have been released at
https://github.com/zjukg/AdaMF-MAT.
Related papers
- Mixture of Modality Knowledge Experts for Robust Multi-modal Knowledge Graph Completion [51.80447197290866]
Multi-modal knowledge graph completion (MMKGC) aims to automatically discover new knowledge triples in the given multi-modal knowledge graphs (MMKGs)
Existing methods tend to focus on crafting elegant entity-wise multi-modal fusion strategies, yet they overlook the utilization of multi-perspective features concealed within the modalities under diverse relational contexts.
We introduce a novel MMKGC framework with Mixture of Modality Knowledge experts (MoMoK) to learn adaptive multi-modal embedding under intricate relational contexts.
arXiv Detail & Related papers (2024-05-27T06:36:17Z) - MyGO: Discrete Modality Information as Fine-Grained Tokens for Multi-modal Knowledge Graph Completion [51.80447197290866]
We introduce MyGO to process, fuse, and augment the fine-grained modality information from MMKGs.
MyGO tokenizes multi-modal raw data as fine-grained discrete tokens and learns entity representations with a cross-modal entity encoder.
Experiments on standard MMKGC benchmarks reveal that our method surpasses 20 of the latest models.
arXiv Detail & Related papers (2024-04-15T05:40:41Z) - NativE: Multi-modal Knowledge Graph Completion in the Wild [51.80447197290866]
We propose a comprehensive framework NativE to achieve MMKGC in the wild.
NativE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities.
We construct a new benchmark called WildKGC with five datasets to evaluate our method.
arXiv Detail & Related papers (2024-03-28T03:04:00Z) - Improving Discriminative Multi-Modal Learning with Large-Scale
Pre-Trained Models [51.5543321122664]
This paper investigates how to better leverage large-scale pre-trained uni-modal models to enhance discriminative multi-modal learning.
We introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA)
arXiv Detail & Related papers (2023-10-08T15:01:54Z) - MACO: A Modality Adversarial and Contrastive Framework for
Modality-missing Multi-modal Knowledge Graph Completion [18.188971531961663]
We propose a modality adversarial and contrastive framework (MACO) to solve the modality-missing problem in MMKGC.
MACO trains a generator and discriminator adversarially to generate missing modality features that can be incorporated into the MMKGC model.
arXiv Detail & Related papers (2023-08-13T06:29:38Z) - VERITE: A Robust Benchmark for Multimodal Misinformation Detection
Accounting for Unimodal Bias [17.107961913114778]
multimodal misinformation is a growing problem on social media platforms.
In this study, we investigate and identify the presence of unimodal bias in widely-used MMD benchmarks.
We introduce a new method -- termed Crossmodal HArd Synthetic MisAlignment (CHASMA) -- for generating realistic synthetic training data.
arXiv Detail & Related papers (2023-04-27T12:28:29Z) - Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal
Sentiment Analysis [96.46952672172021]
Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations.
Model takes two bimodal pairs as input due to known information imbalance among modalities.
arXiv Detail & Related papers (2021-07-28T23:33:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.