GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

• URL: http://arxiv.org/abs/2603.13370v1
• Date: Mon, 09 Mar 2026 16:37:55 GMT
• Title: GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
• Abstract summary: Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals.<n> Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery.<n>We present GraphVLM, a benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning.
• Score: 22.486504107013307
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.

Related papers

• Graph4MM: Weaving Multimodal Learning with Structural Information [52.16646463590474]
  Graphs provide powerful structural information for modeling intra- and inter-modal relationships.<n>Previous works fail to distinguish multi-hop neighbors and treat the graph as a standalone modality.<n>We propose Graph4MM, a graph-based multimodal learning framework.
  arXiv  Detail & Related papers  (2025-10-19T20:13:03Z)
• Graph-MLLM: Harnessing Multimodal Large Language Models for Multimodal Graph Learning [23.089644598166885]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in representing and understanding diverse modalities.<n>Integrating multimodality with structured graph information (i.e., multimodal graphs, MMGs) is essential for real-world applications such as social networks, healthcare, and recommendation systems.<n>Existing MMG learning methods fall into three paradigms based on how they leverage MLLMs.
  arXiv  Detail & Related papers  (2025-06-12T01:44:46Z)
• MLaGA: Multimodal Large Language and Graph Assistant [9.985787670804823]
  Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis.<n>We introduce the Multimodal Large Language and Graph Assistant (MLaGA), an innovative model that adeptly extends LLM capabilities to facilitate reasoning over complex graph structures and multimodal attributes.
  arXiv  Detail & Related papers  (2025-06-03T07:52:00Z)
• Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models [10.813015912529936]
  We introduce the first comprehensive benchmark designed to evaluate and enhance the multi-graph reasoning abilities of Vision-Language Models (VLMs)<n>Our benchmark covers four common graph types-knowledge graphs, flowcharts, mind maps, and route maps-and supports both homogeneous and heterogeneous graph groupings.<n>We evaluate several state-of-the-art VLMs under a multi-dimensional scoring framework that assesses graph parsing, reasoning consistency, and instruction-following accuracy.
  arXiv  Detail & Related papers  (2025-03-27T12:20:37Z)
• UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphs [34.48393396390799]
  We propose a novel cross-domain graph foundation model that enables general representation learning on multimodal graphs.<n>UniGraph2 employs modality-specific encoders alongside a graph neural network (GNN) to learn a unified low-dimensional embedding space.<n>We show that UniGraph2 significantly outperforms state-of-the-art models in tasks such as representation learning, transfer learning, and multimodal generative tasks.
  arXiv  Detail & Related papers  (2025-02-02T14:04:53Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs [61.143381152739046]
  We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach.<n>Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations.<n>We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes.
  arXiv  Detail & Related papers  (2024-06-24T17:59:42Z)
• Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions [31.71954519657729]
  Vision-Language Models (VLMs) are advanced models that can tackle more intricate tasks such as image captioning and visual question answering.<n>Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs.<n>We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible.
  arXiv  Detail & Related papers  (2024-02-20T18:57:34Z)
• Multimodal Graph Learning for Generative Tasks [89.44810441463652]
  Multimodal learning combines multiple data modalities, broadening the types and complexity of data our models can utilize. We propose Multimodal Graph Learning (MMGL), a framework for capturing information from multiple multimodal neighbors with relational structures among them.
  arXiv  Detail & Related papers  (2023-10-11T13:25:03Z)
• Divert More Attention to Vision-Language Object Tracking [87.31882921111048]
  We argue that the lack of large-scale vision-language annotated videos and ineffective vision-language interaction learning motivate us to design more effective vision-language representation for tracking. Particularly, in this paper, we first propose a general attribute annotation strategy to decorate videos in six popular tracking benchmarks, which contributes a large-scale vision-language tracking database with more than 23,000 videos. We then introduce a novel framework to improve tracking by learning a unified-adaptive VL representation, where the cores are the proposed asymmetric architecture search and modality mixer (ModaMixer)
  arXiv  Detail & Related papers  (2023-07-19T15:22:06Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.