MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph

• URL: http://arxiv.org/abs/2605.10120v1
• Date: Mon, 11 May 2026 07:35:55 GMT
• Title: MicroWorld: Empowering Multimodal Large Language Models to Bridge the Microscopic Domain Gap with Multimodal Attribute Graph
• Abstract summary: MicroWorld is a framework that constructs a multimodal attributed property graph from large-scale scientific image--caption corpora.<n>It augments MLLM reasoning at inference time without any domain-specific fine-tuning.<n>On the MicroVQA benchmark, MicroWorld improves the reasoning performance of Qwen3-VL-8B-Instruct by 37.5%.
• Score: 28.472848113791162
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal large language models (MLLMs) show remarkable potential for scientific reasoning, yet their performance in specialized domains such as microscopy remains limited by the scarcity of domain-specific training data and the difficulty of encoding fine-grained expert knowledge into model parameters. To bridge the gap, we introduce MicroWorld, a framework that constructs a multimodal attributed property graph (MAPG) from large-scale scientific image--caption corpora and leverages it to augment MLLM reasoning at inference time without any domain-specific fine-tuning. MicroWorld extracts biomedical entities and relations via scispaCy or LLM-based triplet mining, aligns images and entities in a shared embedding space using Qwen3-VL-Embedding, and assembles a knowledge graph comprising approximately 111K nodes and 346K typed edges spanning eight relation categories. At inference time, a graph-augmented retrieval pipeline matches query entities to the MAPG and injects structured knowledge context into the MLLM prompt. On the MicroVQA benchmark, MicroWorld improves the reasoning performance of Qwen3-VL-8B-Instruct by 37.5%, outperforming GPT-5 by 13.0% to achieve a new state-of-the-art. Furthermore, it yields a 6.0% performance gain on the MicroBench benchmark. Extensive experiments demonstrate the enhanced generalization capability introduced by MicroWorld. A qualitative case study further reveals both the mechanisms through which structured knowledge improves reasoning and the failure modes that point to promising future directions. Code and data are available at https://github.com/ieellee/MicroWorld.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios [58.34124792457706]
  The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
  arXiv  Detail & Related papers  (2026-04-08T12:23:27Z)
• MM-OpenFGL: A Comprehensive Benchmark for Multimodal Federated Graph Learning [33.909733872102656]
  Multimodal-attributed graphs (MMAGs) provide a unified framework for modeling complex relational data by integrating heterogeneous modalities with graph structures.<n> MM-OpenFGL is the first comprehensive benchmark that systematically formalizes the MMFGL paradigm and enables rigorous evaluation.<n> MM-OpenFGL comprises 19 multimodal datasets spanning 7 application domains, 8 simulation strategies capturing modality and topology variations, 6 downstream tasks, and 57 state-of-the-art methods implemented through a modular API.
  arXiv  Detail & Related papers  (2026-01-29T23:59:13Z)
• From Macro to Micro: Benchmarking Microscopic Spatial Intelligence on Molecules via Vision-Language Models [49.40724953627119]
  This paper introduces the concept of Microscopic Spatial Intelligence (MiSI), the capability to perceive and reason about the spatial relationships of invisible microscopic entities.<n>To assess the potential of Vision-Language Models (VLMs) in this domain, we propose a systematic benchmark framework MiSI-Bench.<n>This framework features over 163,000 question-answer pairs and 587,000 images derived from approximately 4,000 molecular structures.
  arXiv  Detail & Related papers  (2025-12-11T18:00:21Z)
• MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model [28.472848113791162]
  MicroVQA++ is a three-stage, large-scale and high-quality microscopy VQA corpus.<n>It bootstraps supervision from expert-validated figure-caption pairs sourced from peer-reviewed articles.<n>HiCQA-Graph is a novel heterogeneous graph over images, captions, and QAs that fuses NLI-based textual entailment, CLIP-based vision-language alignment, and agent signals.
  arXiv  Detail & Related papers  (2025-11-14T15:35:43Z)
• The Next Layer: Augmenting Foundation Models with Structure-Preserving and Attention-Guided Learning for Local Patches to Global Context Awareness in Computational Pathology [23.32822092398391]
  We present EAGLE-Net, a structure-preserving, attention-guided MIL architecture designed to augment prediction and interpretability.<n>We benchmarked it on large pan-cancer datasets, including 3 cancer types for classification (10,260 slides) and 7 cancer types for survival prediction (4,172 slides)
  arXiv  Detail & Related papers  (2025-08-27T14:19:38Z)
• Graph-MLLM: Harnessing Multimodal Large Language Models for Multimodal Graph Learning [23.089644598166885]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in representing and understanding diverse modalities.<n>Integrating multimodality with structured graph information (i.e., multimodal graphs, MMGs) is essential for real-world applications such as social networks, healthcare, and recommendation systems.<n>Existing MMG learning methods fall into three paradigms based on how they leverage MLLMs.
  arXiv  Detail & Related papers  (2025-06-12T01:44:46Z)
• Abstractive Visual Understanding of Multi-modal Structured Knowledge: A New Perspective for MLLM Evaluation [48.462734327375536]
  Multi-modal large language models (MLLMs) incorporate heterogeneous modalities into LLMs, enabling a comprehensive understanding of diverse scenarios and objects.<n>Despite the proliferation of evaluation benchmarks and leaderboards for MLLMs, they predominantly overlook the critical capacity of MLLMs to comprehend world knowledge with structured abstractions that appear in visual form.<n>We propose M3STR, an innovative benchmark grounded in the Multi-Modal Map for STRuctured understanding.<n>Our findings reveal persistent deficiencies in processing abstractive visual information with structured knowledge, thereby charting a pivotal trajectory for advancing MLLMs' holistic reasoning capacities.
  arXiv  Detail & Related papers  (2025-06-02T04:00:35Z)
• MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research [57.61445960384384]
  MicroVQA consists of 1,042 multiple-choice questions (MCQs) curated by biology experts across diverse microscopy modalities.<n> Benchmarking on state-of-the-art MLLMs reveal a peak performance of 53%.<n>Expert analysis of chain-of-thought responses shows perception errors are the most frequent, followed by knowledge errors and then overgeneralization errors.
  arXiv  Detail & Related papers  (2025-03-17T17:33:10Z)
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks.<n>Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales.<n>We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• When Graph meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graphs Learning [28.555788831123447]
  Multimodal Attributed Graphs (MAGs) are ubiquitous in real-world applications, encompassing extensive knowledge through multimodal attributes attached to nodes.<n>Despite its potential to advance diverse research fields like social networks and e-commerce, MAG representation learning (MAGRL) remains underexplored.<n>We first propose MAGB, a comprehensive MAG benchmark dataset, featuring curated graphs from various domains with both textual and visual attributes.
  arXiv  Detail & Related papers  (2024-10-11T13:24:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.