When Graph meets Multimodal: Benchmarking on Multimodal Attributed Graphs Learning

• URL: http://arxiv.org/abs/2410.09132v1
• Date: Fri, 11 Oct 2024 13:24:57 GMT
• Title: When Graph meets Multimodal: Benchmarking on Multimodal Attributed Graphs Learning
• Authors: Hao Yan, Chaozhuo Li, Zhigang Yu, Jun Yin, Ruochen Liu, Peiyan Zhang, Weihao Han, Mingzheng Li, Zhengxin Zeng, Hao Sun, Weiwei Deng, Feng Sun, Qi Zhang, Senzhang Wang,
• Abstract summary: Multimodal attributed graphs (MAGs) are prevalent in various real-world scenarios and generally contain two kinds of knowledge. Recent advancements in Pre-trained Language/Vision models (PLMs/PVMs) and Graph neural networks (GNNs) have facilitated effective learning on MAGs. We propose Multimodal Attribute Graph Benchmark (MAGB), a comprehensive and diverse collection of challenging benchmark datasets for MAGs.
• Score: 36.6581535146878
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal attributed graphs (MAGs) are prevalent in various real-world scenarios and generally contain two kinds of knowledge: (a) Attribute knowledge is mainly supported by the attributes of different modalities contained in nodes (entities) themselves, such as texts and images. (b) Topology knowledge, on the other hand, is provided by the complex interactions posed between nodes. The cornerstone of MAG representation learning lies in the seamless integration of multimodal attributes and topology. Recent advancements in Pre-trained Language/Vision models (PLMs/PVMs) and Graph neural networks (GNNs) have facilitated effective learning on MAGs, garnering increased research interest. However, the absence of meaningful benchmark datasets and standardized evaluation procedures for MAG representation learning has impeded progress in this field. In this paper, we propose Multimodal Attribute Graph Benchmark (MAGB)}, a comprehensive and diverse collection of challenging benchmark datasets for MAGs. The MAGB datasets are notably large in scale and encompass a wide range of domains, spanning from e-commerce networks to social networks. In addition to the brand-new datasets, we conduct extensive benchmark experiments over MAGB with various learning paradigms, ranging from GNN-based and PLM-based methods, to explore the necessity and feasibility of integrating multimodal attributes and graph topology. In a nutshell, we provide an overview of the MAG datasets, standardized evaluation procedures, and present baseline experiments. The entire MAGB project is publicly accessible at https://github.com/sktsherlock/ATG.

Related papers

• UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs [34.48393396390799]
  We propose a novel cross-domain graph foundation model that enables general representation learning on multimodal graphs. UniGraph2 employs modality-specific encoders alongside a graph neural network (GNN) to learn a unified low-dimensional embedding space. We show that UniGraph2 significantly outperforms state-of-the-art models in tasks such as representation learning, transfer learning, and multimodal generative tasks.
  arXiv  Detail & Related papers  (2025-02-02T14:04:53Z)
• Mamba-Based Graph Convolutional Networks: Tackling Over-smoothing with Selective State Space [33.677431350509224]
  We introduce MbaGCN, a novel graph convolutional architecture that draws inspiration from the Mamba paradigm. MbaGCN presents a new backbone for GNNs, consisting of three key components: the Message Aggregation Layer, the Selective State Space Transition Layer, and the Node State Prediction Layer.
  arXiv  Detail & Related papers  (2025-01-26T09:09:44Z)
• Multimodal Graph Benchmark [36.75510196380185]
  Multimodal Graph Benchmark (MM-GRAPH) is first comprehensive multi-modal graph benchmark that incorporates both textual and visual information. MM-GRAPH consists of five graph learning datasets of various scales that are appropriate for different learning tasks. MM-GRAPH aims to foster research on multimodal graph learning and drive the development of more advanced and robust graph learning algorithms.
  arXiv  Detail & Related papers  (2024-06-24T05:14:09Z)
• Tokenization, Fusion, and Augmentation: Towards Fine-grained Multi-modal Entity Representation [51.80447197290866]
  Multi-modal knowledge graph completion (MMKGC) aims to discover unobserved knowledge from given knowledge graphs. Existing MMKGC methods usually extract multi-modal features with pre-trained models. We introduce a novel framework MyGO to tokenize, fuse, and augment the fine-grained multi-modal representations of entities.
  arXiv  Detail & Related papers  (2024-04-15T05:40:41Z)
• NativE: Multi-modal Knowledge Graph Completion in the Wild [51.80447197290866]
  We propose a comprehensive framework NativE to achieve MMKGC in the wild. NativE proposes a relation-guided dual adaptive fusion module that enables adaptive fusion for any modalities. We construct a new benchmark called WildKGC with five datasets to evaluate our method.
  arXiv  Detail & Related papers  (2024-03-28T03:04:00Z)
• Noise-powered Multi-modal Knowledge Graph Representation Framework [52.95468915728721]
  The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph representation learning framework. We propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking. Our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility.
  arXiv  Detail & Related papers  (2024-03-11T15:48:43Z)
• Learning on Multimodal Graphs: A Survey [6.362513821299131]
  Multimodal data pervades various domains, including healthcare, social media, and transportation. multimodal graph learning (MGL) is essential for successful artificial intelligence (AI) applications.
  arXiv  Detail & Related papers  (2024-02-07T23:50:00Z)
• Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation [68.13453771001522]
  We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings. We conduct extensive experiments and evaluate our model on large-scale real-world data.
  arXiv  Detail & Related papers  (2023-06-14T13:07:48Z)
• Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion [112.27103169303184]
  Multimodal Knowledge Graphs (MKGs) organize visual-text factual knowledge. MKGformer can obtain SOTA performance on four datasets of multimodal link prediction, multimodal RE, and multimodal NER.
  arXiv  Detail & Related papers  (2022-05-04T23:40:04Z)
• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification [43.35966675372692]
  We show how to train deep networks and build the network architecture. In particular, we show different fusion strategies as well as how to train deep networks and build the network architecture. Our framework is not only limited to pixel-wise classification tasks but also applicable to spatial information modeling with convolutional neural networks (CNNs)
  arXiv  Detail & Related papers  (2020-08-12T17:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.