Spatial Knowledge Graph-Guided Multimodal Synthesis

• URL: http://arxiv.org/abs/2505.22633v1
• Date: Wed, 28 May 2025 17:50:21 GMT
• Title: Spatial Knowledge Graph-Guided Multimodal Synthesis
• Authors: Yida Xue, Zhen Bi, Jinnan Yang, Jungang Lou, Huajun Chen, Ningyu Zhang,
• Abstract summary: We introduce SKG2Data, a novel multimodal synthesis approach guided by spatial knowledge graphs.<n>In this work, we introduce SKG2Data, a novel multimodal data synthesis approach guided by spatial knowledge graphs.
• Score: 41.91964971544324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in multimodal large language models (MLLMs) have significantly enhanced their capabilities; however, their spatial perception abilities remain a notable limitation. To address this challenge, multimodal data synthesis offers a promising solution. Yet, ensuring that synthesized data adhere to spatial common sense is a non-trivial task. In this work, we introduce SKG2Data, a novel multimodal synthesis approach guided by spatial knowledge graphs, grounded in the concept of knowledge-to-data generation. SKG2Data automatically constructs a Spatial Knowledge Graph (SKG) to emulate human-like perception of spatial directions and distances, which is subsequently utilized to guide multimodal data synthesis. Extensive experiments demonstrate that data synthesized from diverse types of spatial knowledge, including direction and distance, not only enhance the spatial perception and reasoning abilities of MLLMs but also exhibit strong generalization capabilities. We hope that the idea of knowledge-based data synthesis can advance the development of spatial intelligence.

Related papers

• Can LLMs Learn to Map the World from Local Descriptions? [50.490593949836146]
  This study investigates whether Large Language Models (LLMs) can construct coherent global spatial cognition.<n> Experiments conducted in a simulated urban environment demonstrate that LLMs exhibit latent representations aligned with real-world spatial distributions.
  arXiv  Detail & Related papers  (2025-05-27T08:22:58Z)
• OmniGeo: Towards a Multimodal Large Language Models for Geospatial Artificial Intelligence [51.0456395687016]
  multimodal large language models (LLMs) have opened new frontiers in artificial intelligence.<n>We propose a MLLM (OmniGeo) tailored to geospatial applications.<n>By combining the strengths of natural language understanding and spatial reasoning, our model enhances the ability of instruction following and the accuracy of GeoAI systems.
  arXiv  Detail & Related papers  (2025-03-20T16:45:48Z)
• Conservation-informed Graph Learning for Spatiotemporal Dynamics Prediction [84.26340606752763]
  In this paper, we introduce the conservation-informed GNN (CiGNN), an end-to-end explainable learning framework.<n>The network is designed to conform to the general symmetry conservation law via symmetry where conservative and non-conservative information passes over a multiscale space by a latent temporal marching strategy.<n>Results demonstrate that CiGNN exhibits remarkable baseline accuracy and generalizability, and is readily applicable to learning for prediction of varioustemporal dynamics.
  arXiv  Detail & Related papers  (2024-12-30T13:55:59Z)
• Spherinator and HiPSter: Representation Learning for Unbiased Knowledge Discovery from Simulations [0.0]
  We describe a new, unbiased, and machine learning based approach to obtain useful scientific insights from a broad range of simulations. Our concept is based on applying nonlinear dimensionality reduction to learn compact representations of the data in a low-dimensional space. We present a prototype using a rotational invariant hyperspherical variational convolutional autoencoder, utilizing a power distribution in the latent space, and trained on galaxies from IllustrisTNG simulation.
  arXiv  Detail & Related papers  (2024-06-06T07:34:58Z)
• Deep Learning for Spatiotemporal Big Data: A Vision on Opportunities and Challenges [4.497634148674422]
  Intemporal big data can foster new opportunities to solve problems that have not been possible before. The distinctive characteristics of big data pose new challenges for deep learning technologies.
  arXiv  Detail & Related papers  (2023-10-30T19:12:51Z)
• Approach to Data Science with Multiscale Information Theory [0.0]
  Data Science is a multidisciplinary field that plays a crucial role in extracting valuable insights from large and intricate datasets. Within the realm of Data Science, two fundamental components are Information Theory (IT) and Statistical Mechanics (SM) In this paper, we apply this data science framework to a large and intricate mechanical system composed of particles.
  arXiv  Detail & Related papers  (2023-05-23T01:08:50Z)
• Semantic Segmentation of Vegetation in Remote Sensing Imagery Using Deep Learning [77.34726150561087]
  We propose an approach for creating a multi-modal and large-temporal dataset comprised of publicly available Remote Sensing data. We use Convolutional Neural Networks (CNN) models that are capable of separating different classes of vegetation.
  arXiv  Detail & Related papers  (2022-09-28T18:51:59Z)
• Dominant motion identification of multi-particle system using deep learning from video [0.0]
  In this work, we provide a deep-learning framework that extracts relevant information from real-world videos of highly systems. We demonstrate this approach on videos of confined multi-agent/particle systems of ants, termites, fishes. Furthermore, we explore how these seemingly diverse systems have predictable underlying behavior.
  arXiv  Detail & Related papers  (2021-04-26T17:10:56Z)
• Synthetic Data: Opening the data floodgates to enable faster, more directed development of machine learning methods [96.92041573661407]
  Many ground-breaking advancements in machine learning can be attributed to the availability of a large volume of rich data. Many large-scale datasets are highly sensitive, such as healthcare data, and are not widely available to the machine learning community. Generating synthetic data with privacy guarantees provides one such solution.
  arXiv  Detail & Related papers  (2020-12-08T17:26:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.