Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models

• URL: http://arxiv.org/abs/2605.17669v1
• Date: Sun, 17 May 2026 22:02:35 GMT
• Title: Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models
• Abstract summary: We introduce a new knowledge graph in the domain of French cultural heritage, WJoconde, which is distinguished by its multimodality.<n>We propose a new framework for extending cultural heritage Knowledge Graphs using multi-modal approaches.<n>Our results show that by integrating the rich text and image information in cultural heritage data, we can efficiently enhance KGs with high reliability.
• Score: 3.3313496283191912
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The preservation and interpretation of cultural heritage increasingly rely on digital technologies, among which Knowledge Graphs (KGs) stand out for their ability to structure vast amounts of data. However, the construction and expansion of these KGs often face challenges due to the diverse and complex nature of cultural heritage information. In this paper, we propose a novel approach for extending KG resources in the domain of cultural heritage, which we applied to French data. First, we introduce a new knowledge graph in the domain of French cultural heritage, WJoconde, which is distinguished by its multimodality as it integrates both textual and image information of the entities. We further introduce three variants of WJoconde to facilitate downstream research, such as Knowledge Graph Completion (KGC). We also built a comprehensive benchmark for KGC methods on our dataset. Second, we propose a new framework for extending cultural heritage KGs using multi-modal approaches leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs), which includes automated data extraction from unstructured resources combined with a special validation pipeline for grounding the output of both models, to further extend WJoconde. Our results show that by integrating the rich text and image information in cultural heritage data, we can efficiently enhance KGs with high reliability. We open-source all code and benchmark datasets with text and images, as well as the original data with an interactive access point

Related papers

• Competency Questions as Executable Plans: a Controlled RAG Architecture for Cultural Heritage Storytelling [0.12891210250935145]
  We propose a novel neuro-symbolic architecture grounded in Knowledge Graphs to generate narratives.<n>A key novelty of our approach is the repurposing of competency questions (CQs) into run-time executable narrative plans.<n>We validate this architecture using a new resource: the Live Aid KG, a multimodal dataset aligning 1985 concert data with the Music Meta Ontology.
  arXiv  Detail & Related papers  (2026-04-02T21:54:33Z)
• Grounding Multilingual Multimodal LLMs With Cultural Knowledge [48.95126394270723]
  We propose a data-centric approach that grounds MLLMs in cultural knowledge.<n>CulturalGround comprises 22 million high-quality, culturally-rich VQA pairs spanning 42 countries and 39 languages.<n>We train an open-source MLLM CulturalPangea on CulturalGround, interleaving standard multilingual instruction-tuning data to preserve general abilities.
  arXiv  Detail & Related papers  (2025-08-10T16:24:11Z)
• GLTW: Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completion [52.026016846945424]
  We propose a new method called GLTW, which encodes the structural information of KGs and merges it with Large Language Models.<n>Specifically, we introduce an improved Graph Transformer (iGT) that effectively encodes subgraphs with both local and global structural information.<n>Also, we develop a subgraph-based multi-classification training objective, using all entities within KG as classification objects, to boost learning efficiency.
  arXiv  Detail & Related papers  (2025-02-17T06:02:59Z)
• Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking [48.21982147529661]
  This paper introduces a novel approach for massively multicultural knowledge acquisition. Our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI.
  arXiv  Detail & Related papers  (2024-02-14T18:16:54Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• Geolocation of Cultural Heritage using Multi-View Knowledge Graph Embedding [18.822364073669583]
  We present a framework for ingesting knowledge about tangible cultural heritage entities. We also propose a learning model for estimating the relative distance between a pair of cultural heritage entities.
  arXiv  Detail & Related papers  (2022-09-08T08:32:34Z)
• Heri-Graphs: A Workflow of Creating Datasets for Multi-modal Machine Learning on Graphs of Heritage Values and Attributes with Social Media [7.318997639507268]
  Values (why to conserve) and Attributes (what to conserve) are essential concepts of cultural heritage. Recent studies have been using social media to map values and attributes conveyed by public to cultural heritage. This study presents a methodological workflow for constructing such multi-modal datasets using posts and images on Flickr.
  arXiv  Detail & Related papers  (2022-05-16T09:45:45Z)
• Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis [96.53859361560505]
  We propose a knowledge graph augmented network (KGAN) to incorporate external knowledge with explicitly syntactic and contextual information. KGAN captures the sentiment feature representations from multiple perspectives, i.e., context-, syntax- and knowledge-based. Experiments on three popular ABSA benchmarks demonstrate the effectiveness and robustness of our KGAN.
  arXiv  Detail & Related papers  (2022-01-13T08:25:53Z)
• Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation [49.89831914386982]
  We propose a unified pre-trained language model (PLM) for all forms of text, including unstructured text, semi-structured text, and well-structured text. Our approach outperforms the pre-training of plain text using only 1/4 of the data.
  arXiv  Detail & Related papers  (2021-09-02T16:05:24Z)
• Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training [22.534866015730664]
  We verbalize the entire English Wikidata KG. We show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora.
  arXiv  Detail & Related papers  (2020-10-23T22:14:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.