Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifacts

• URL: http://arxiv.org/abs/2502.14865v1
• Date: Thu, 20 Feb 2025 18:59:51 GMT
• Title: Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifacts
• Authors: Sara Ghaboura, Ketan More, Ritesh Thawkar, Wafa Alghallabi, Omkar Thawakar, Fahad Shahbaz Khan, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer,
• Abstract summary: TimeTravel is a benchmark of 10,250 expert-verified samples spanning 266 distinct cultures across 10 major historical regions. TimeTravel is designed for AI-driven analysis of manuscripts, artworks, inscriptions, and archaeological discoveries. We evaluate contemporary AI models on TimeTravel, highlighting their strengths and identifying areas for improvement.
• Score: 65.90535970515266
• License:
• Abstract: Understanding historical and cultural artifacts demands human expertise and advanced computational techniques, yet the process remains complex and time-intensive. While large multimodal models offer promising support, their evaluation and improvement require a standardized benchmark. To address this, we introduce TimeTravel, a benchmark of 10,250 expert-verified samples spanning 266 distinct cultures across 10 major historical regions. Designed for AI-driven analysis of manuscripts, artworks, inscriptions, and archaeological discoveries, TimeTravel provides a structured dataset and robust evaluation framework to assess AI models' capabilities in classification, interpretation, and historical comprehension. By integrating AI with historical research, TimeTravel fosters AI-powered tools for historians, archaeologists, researchers, and cultural tourists to extract valuable insights while ensuring technology contributes meaningfully to historical discovery and cultural heritage preservation. We evaluate contemporary AI models on TimeTravel, highlighting their strengths and identifying areas for improvement. Our goal is to establish AI as a reliable partner in preserving cultural heritage, ensuring that technological advancements contribute meaningfully to historical discovery. Our code is available at: \url{https://github.com/mbzuai-oryx/TimeTravel}.

Related papers

• Grand Challenges in Immersive Technologies for Cultural Heritage [6.678822458675665]
  The integration of immersive technologies has transformed how cultural heritage is presented. The adoption of these technologies also brings a range of challenges and potential risks.
  arXiv  Detail & Related papers  (2024-12-03T21:39:01Z)
• O1 Replication Journey: A Strategic Progress Report -- Part 1 [52.062216849476776]
  This paper introduces a pioneering approach to artificial intelligence research, embodied in our O1 Replication Journey. Our methodology addresses critical challenges in modern AI research, including the insularity of prolonged team-based projects. We propose the journey learning paradigm, which encourages models to learn not just shortcuts, but the complete exploration process.
  arXiv  Detail & Related papers  (2024-10-08T15:13:01Z)
• CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge [69.82940934994333]
  We introduce CulturalTeaming, an interactive red-teaming system that leverages human-AI collaboration to build challenging evaluation dataset. Our study reveals that CulturalTeaming's various modes of AI assistance support annotators in creating cultural questions. CULTURALBENCH-V0.1 is a compact yet high-quality evaluation dataset with users' red-teaming attempts.
  arXiv  Detail & Related papers  (2024-04-10T00:25:09Z)
• CHisIEC: An Information Extraction Corpus for Ancient Chinese History [12.41912979618724]
  We present the Chinese Historical Information Extraction Corpus''(CHis IEC) dataset. CHis IEC is a meticulously curated dataset designed to develop and evaluate NER and RE tasks. The dataset encompasses four distinct entity types and twelve relation types, resulting in a meticulously labeled dataset.
  arXiv  Detail & Related papers  (2024-03-22T10:12:10Z)
• Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking [48.21982147529661]
  This paper introduces a novel approach for massively multicultural knowledge acquisition. Our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI.
  arXiv  Detail & Related papers  (2024-02-14T18:16:54Z)
• Insightful analysis of historical sources at scales beyond human capabilities using unsupervised Machine Learning and XAI [4.593752628215474]
  This study centers on the evolution of knowledge within the Sacrobosco Collection' -- a digitized collection of 359 early modern printed editions of textbooks on astronomy at European universities between 1472 and 1650. An ML based analysis of these tables helps to unveil important facets of the piecing-temporal evolution of knowledge and innovation in the field of mathematical astronomy in the period.
  arXiv  Detail & Related papers  (2023-10-13T13:22:05Z)
• Exploration with Principles for Diverse AI Supervision [88.61687950039662]
  Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. We propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data.
  arXiv  Detail & Related papers  (2023-10-13T07:03:39Z)
• ScrollTimes: Tracing the Provenance of Paintings as a Window into History [35.605930297790465]
  The study of cultural artifact provenance, tracing ownership and preservation, holds significant importance in archaeology and art history. In collaboration with art historians, we examined the handscroll, a traditional Chinese painting form that provides a rich source of historical data. We present a three-tiered methodology encompassing artifact, contextual, and provenance levels, designed to create a "Biography" for handscroll.
  arXiv  Detail & Related papers  (2023-06-15T03:38:09Z)
• Learning Robust Real-Time Cultural Transmission without Human Data [82.05222093231566]
  We provide a method for generating zero-shot, high recall cultural transmission in artificially intelligent agents. Our agents succeed at real-time cultural transmission from humans in novel contexts without using any pre-collected human data. This paves the way for cultural evolution as an algorithm for developing artificial general intelligence.
  arXiv  Detail & Related papers  (2022-03-01T19:32:27Z)
• The computerization of archaeology: survey on AI techniques [6.985152632198481]
  This paper analyses the application of artificial intelligence techniques to various areas of archaeology and more specifically: a) The use of software tools as a creative stimulus for the organization of exhibitions;. The classification of fragments found in archaeological excavations and for the reconstruction of ceramics;. The cataloguing and study of human remains to understand the social and historical context of belonging;. The design of a study for the exploration of marine archaeological sites, located at depths that cannot be reached by man, through the construction of a freely explorable 3D version.
  arXiv  Detail & Related papers  (2020-05-05T17:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.