MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

• URL: http://arxiv.org/abs/2607.15592v1
• Date: Fri, 17 Jul 2026 03:39:43 GMT
• Title: MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion
• Abstract summary: Multimodal Knowledge Graph Completion (MKGC) requires inferring missing entities from structural, textual, and visual cues.<n>Existing diffusion-based MKGC methods usually denoise directly on raw multimodal features.<n>We propose MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts.
• Score: 33.838877596911445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Knowledge Graph Completion (MKGC) requires inferring missing entities from structural, textual, and visual cues. Existing diffusion-based MKGC methods usually denoise directly on raw multimodal features. Such a design forces the denoiser to simultaneously perform relation-dependent cue selection, cross-modal semantic alignment, and structure-aware entity generation, which introduces noisy and semantically inconsistent conditions for diffusion and consequently leads to suboptimal completion performance. To address this limitation, we propose MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts (MGDT), a novel MKGC framework built on an align-then-diffuse paradigm. MGDT first employs a Relation-Adaptive Semantic Routing Mixture-of-Experts (RASR-MoE) module to select relation-relevant multimodal semantic transformation paths and suppress irrelevant modality interference. MGDT then uses a frozen Multimodal Large Language Model (MLLM) as a semantic anchor to align the routed multimodal representations into a unified latent space and reduce cross-modal semantic heterogeneity. Finally, a Knowledge Graph Diffusion Transformer (KGDT) performs graph-conditioned denoising generation in the aligned space to produce the missing entity representation. Experiments on three benchmark datasets show that MGDT consistently outperforms strong baselines.

Related papers

• Prototype-Anchored Generalized Manifold Regression for Unknown-Domain Object Detection [58.25418970608328]
  We study Single-Domain Generalized Object Detection (Single-DGOD), which aims to transfer a detector trained on a single source domain to multiple unseen domains.<n>Existing methods mainly rely on simulation-driven strategies, such as data augmentation or textual prompts, to enlarge the training distribution.<n>We propose Manifold Regression with Visual-Text Dual Chain-of-Thought (MR-DCoT), which formulates unknown-domain generalization as a manifold regression problem.
  arXiv  Detail & Related papers  (2026-07-08T09:27:15Z)
• SAMA: Semantic Anchor-aligned Augmentation for Unified Low-Resource Multimodal Information Extraction [12.379572218573365]
  We introduce Semantic Anchor-aligned Multimodal Augmentation (SAMA), a unified framework for generating high-fidelity, task-aware synthetic data.<n>SAMA constructs structured semantic anchors from ground-truth labels to guide a Collaborative Multi-Experts Multimodal Large Language Model.<n>SAMA consistently outperforms state-of-the-art augmentation baselines under both fully supervised and low-resource settings.
  arXiv  Detail & Related papers  (2026-06-17T07:43:33Z)
• SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs [38.400117494307615]
  Multimodal-attributed graphs (MAGs) couple graph topology with node semantics from text, images, and other modalities.<n>We propose SMGFM, a spectral multimodal graph pretraining framework that decomposes each modality-specific node signal into graph-frequency bands.<n>SMGFM achieves state-of-the-art performance across graph-level and modality-level tasks.
  arXiv  Detail & Related papers  (2026-06-11T03:54:35Z)
• Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMs [84.3271821505699]
  Chain of Modality (CoM) is an agentic framework that transitions multimodal fusion from passive concatenation to dynamic orchestration.<n> operating in either a training-free or a data-efficient SFT setting, CoM achieves robust and consistent generalization across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-04-16T01:21:14Z)
• MAny: Merge Anything for Multimodal Continual Instruction Tuning [52.50936513604062]
  textbfMAny (textbfMAny) is a framework that merges task-specific knowledge through textbfCross-modal textbfProjection textbfMerging.<n>textbfLow-rank textbfParameter textbfMerging (textbfLPM)
  arXiv  Detail & Related papers  (2026-04-15T15:57:23Z)
• Diffusion-CAM: Faithful Visual Explanations for dMLLMs [8.78213424930081]
  Diffusion-CAM is the first interpretability method specifically tailored for dMLLMs.<n>We derive raw activation maps by differentiably probing intermediate representations in the backbone.<n>Our experiments demonstrate that Diffusion-CAM significantly outperforms SoTA methods in both localization accuracy and visual fidelity.
  arXiv  Detail & Related papers  (2026-04-13T05:14:18Z)
• Parameter-Efficient Modality-Balanced Symmetric Fusion for Multimodal Remote Sensing Semantic Segmentation [8.840077295284393]
  MoBaNet is a parameter-efficient and modality-balanced symmetric fusion framework.<n>Built upon a largely frozen VFM backbone, MoBaNet adopts a symmetric dual-stream architecture to preserve generalizable representations.<n>Experiments on the ISPRS Vaihingen and Potsdam benchmarks demonstrate that MoBaNet achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-18T13:23:58Z)
• Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach [99.80480649258557]
  DiTFuse is an instruction-driven framework that performs semantics-aware fusion within a single model.<n>Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention.
  arXiv  Detail & Related papers  (2025-12-08T05:04:54Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• MAGE: Multimodal Alignment and Generation Enhancement via Bridging Visual and Semantic Spaces [23.447713697204225]
  MAGE is a novel framework that bridges the semantic spaces of vision and text through an innovative alignment mechanism.<n>We employ a training strategy that combines cross-entropy and mean squared error, significantly enhancing the alignment effect.<n>Our proposed multimodal large model architecture, MAGE, achieved significantly better performance compared to similar works across various evaluation benchmarks.
  arXiv  Detail & Related papers  (2025-07-29T12:17:46Z)
• Multimodal Latent Language Modeling with Next-Token Diffusion [111.93906046452125]
  Multimodal generative models require a unified approach to handle both discrete data (e.g., text and code) and continuous data (e.g., image, audio, video)<n>We propose Latent Language Modeling (LatentLM), which seamlessly integrates continuous and discrete data using causal Transformers.
  arXiv  Detail & Related papers  (2024-12-11T18:57:32Z)
• Modality Prompts for Arbitrary Modality Salient Object Detection [57.610000247519196]
  This paper delves into the task of arbitrary modality salient object detection (AM SOD) It aims to detect salient objects from arbitrary modalities, eg RGB images, RGB-D images, and RGB-D-T images. A novel modality-adaptive Transformer (MAT) will be proposed to investigate two fundamental challenges of AM SOD.
  arXiv  Detail & Related papers  (2024-05-06T11:02:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.