ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset

• URL: http://arxiv.org/abs/2606.09648v1
• Date: Mon, 08 Jun 2026 15:40:57 GMT
• Title: ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset
• Abstract summary: We present ArtiFact, a multi-modal cultural heritage dataset of 651045 museum records.<n>For cross-modal error detection, we introduce a curated taxonomy of seven error categories injected into 130209 records.<n>For semantic query processing, we show that current systems struggle with queries involving cultural proximity, ambiguous object types, and historically contingent terminology.
• Score: 7.868559688632956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-modal data management has emerged as a central research topic in the database community, spanning data integration, semantic query processing, and data quality assessment. Despite this growing interest, the community lacks large-scale, real-world datasets combining tables, text, and images. We present ArtiFact, a multi-modal cultural heritage dataset of 651045 museum records collected from the Metropolitan Museum of Art, the Art Institute of Chicago, and the Rijksmuseum. We demonstrate the utility of ArtiFact through two downstream tasks. For cross-modal error detection, we introduce a curated taxonomy of seven error categories injected into 130209 records and show that reliably detecting subtle domain-specific errors such as material anachronisms and temporal shifts remain an open challenge. For semantic query processing, we show that current systems struggle with queries involving cultural proximity, ambiguous object types, and historically contingent terminology. Our results position ArtiFact as a challenging benchmark for multi-modal data management research.

Related papers

• On the Cultural Anachronism and Temporal Reasoning in Vision Language Models [35.132248635251266]
  Vision-Language Models (VLMs) are increasingly applied to cultural heritage materials.<n>This work identifies a fundamental issue in how these models interpret historical artifacts.<n>We define this phenomenon as cultural anachronism, the tendency to misinterpret historical objects using temporally inappropriate concepts.
  arXiv  Detail & Related papers  (2026-05-14T16:58:16Z)
• BIRD: A Museum Open Dataset Combining Behavior Patterns and Identity Types to Better Model Visitors' Experience [0.0]
  We conducted a study aimed at obtaining comprehensive and detailed information about visitors, their visit experience and their feedback.<n>We built an open dataset combining contextual data (demographic data, preferences, visiting habits, motivations, social context) and behavioral data (spatiotemporal trajectories, gaze data)<n>Our analysis made it possible to re-enact visitor identities combining the majority of characteristics found in the literature and to reproduce the Veron and Levasseur profiles.
  arXiv  Detail & Related papers  (2025-12-29T07:44:32Z)
• MUSEKG: A Knowledge Graph Over Museum Collections [19.587385754644256]
  MuseKG is an end-to-end knowledge-graph framework for museum information systems.<n>It unifies structured and unstructured museum data through symbolic-neural integration.
  arXiv  Detail & Related papers  (2025-11-20T03:23:36Z)
• Bridging the Data Provenance Gap Across Text, Speech and Video [67.72097952282262]
  We conduct the largest and first-of-its-kind longitudinal audit across modalities of popular text, speech, and video datasets.<n>Our manual analysis covers nearly 4000 public datasets between 1990-2024, spanning 608 languages, 798 sources, 659 organizations, and 67 countries.<n>We find that multimodal machine learning applications have overwhelmingly turned to web-crawled, synthetic, and social media platforms, such as YouTube, for their training sets.
  arXiv  Detail & Related papers  (2024-12-19T01:30:19Z)
• Understanding Museum Exhibits using Vision-Language Reasoning [52.35301212718003]
  Museums serve as repositories of cultural heritage and historical artifacts from diverse epochs, civilizations, and regions.<n> domain-specific models are essential for interactive query resolution and gaining historical insights.<n>We collect and curate a large-scale dataset of 65M images and 200M question-answer pairs for exhibits from all around the world.
  arXiv  Detail & Related papers  (2024-12-02T10:54:31Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• 3AM: An Ambiguity-Aware Multi-Modal Machine Translation Dataset [90.95948101052073]
  We introduce 3AM, an ambiguity-aware MMT dataset comprising 26,000 parallel sentence pairs in English and Chinese. Our dataset is specifically designed to include more ambiguity and a greater variety of both captions and images than other MMT datasets. Experimental results show that MMT models trained on our dataset exhibit a greater ability to exploit visual information than those trained on other MMT datasets.
  arXiv  Detail & Related papers  (2024-04-29T04:01:30Z)
• SGED: A Benchmark dataset for Performance Evaluation of Spiking Gesture Emotion Recognition [12.396844568607522]
  We label a new homogeneous multimodal gesture emotion recognition dataset based on the analysis of the existing data sets. We propose a pseudo dual-flow network based on this dataset, and verify the application potential of this dataset in the affective computing community.
  arXiv  Detail & Related papers  (2023-04-28T09:32:09Z)
• Few-shot Partial Multi-view Learning [103.33865779721458]
  We propose a new task called few-shot partial multi-view learning. It focuses on overcoming the negative impact of the view-missing issue in the low-data regime. We conduct extensive experiments to evaluate our method.
  arXiv  Detail & Related papers  (2021-05-05T13:34:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.