EduArt: An educational-level benchmark for evaluating art history knowledge in large language models

• URL: http://arxiv.org/abs/2607.02007v1
• Date: Thu, 02 Jul 2026 10:43:06 GMT
• Title: EduArt: An educational-level benchmark for evaluating art history knowledge in large language models
• Abstract summary: Existing art-focused evaluations rely on synthetic questions and rarely report item-level properties.<n>This paper introduces EduArt, an educational-level benchmark for art-historical knowledge and visual reasoning in multimodal LLMs.
• Score: 0.764671395172401
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models now score near ceiling on general benchmarks, but these aggregate measures reveal little about how models behave within single disciplines. Existing art-focused evaluations rely on synthetic questions and rarely report item-level properties. This paper introduces EduArt, an educational-level benchmark for art-historical knowledge and visual reasoning in multimodal LLMs. EduArt comprises 871 human-authored questions from Italian secondary-school exercises and US Advanced Placement Art History exams, spanning two languages and seven formats from multiple choice to in-text word placement and error identification. Twelve models from six provider families were evaluated under a default answer-only condition and a motivation condition requiring written justification, and characterized using Classical Test Theory and a logistic regression isolating the effects of format, language, image presence, and model. The benchmark showed strong psychometric properties (mean discrimination 0.514, 82.3 percent good discriminators), while multiple-choice accuracy saturated near ceiling for six models, showing recognition formats alone cannot distinguish frontier models. Format was a strong independent predictor of accuracy: models exceeding 94 percent on multiple choice fell to 23.9 percent on open completion (Claude Opus 4.6) and 6.2 percent on error identification (Claude Sonnet 4.6). The motivation condition changed accuracy in a predominantly negative, family-dependent direction. These dissociations indicate that art-historical knowledge and the ability to deploy it are distinct capabilities, and that single-format benchmarks overestimate what models can reliably do. Mapping this capability profile is a precondition for responsible use of multimodal LLMs in art-historical scholarship, where tasks demand producing and manipulating content rather than selecting from fixed options.

Related papers

• Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models [0.0]
  We present a systematic zero-shot evaluation of 41 open-weight language models spanning 15 families and the 135M--9B parameter range.<n>We analyze confidence calibration, robustness to realistic input perturbations, statistical reliability of model rankings, deployment efficiency, and benchmark saturation.<n>These findings provide practical guidance for selecting and evaluating open-weight language models for intent classification.
  arXiv  Detail & Related papers  (2026-07-29T19:42:33Z)
• Revisiting Northrop Frye's Four Myths Theory with Large Language Models [0.0]
  Northrop Frye's theory of four fundamental narrative genres has profoundly influenced literary criticism.<n>We present a new character function framework that complements pattern-based analysis.<n>We derive four universal character functions (protagonist, mentor, antagonist, companion) by mapping them to Jung's psychic structure components.
  arXiv  Detail & Related papers  (2026-02-17T16:02:52Z)
• AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models [0.0]
  AA- Omniscience is a benchmark designed to measure factual recall and knowledge calibration across 6,000 questions.<n>The evaluation measures a model's Omniscience Index, a bounded metric (-100 to 100) measuring factual recall.<n>Results reveal persistent factuality and calibration weaknesses across frontier models.
  arXiv  Detail & Related papers  (2025-11-17T06:27:16Z)
• DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation [111.94720088481614]
  Can multimodal generative models effectively produce content given dialectal textual input?<n>We construct a new large-scale benchmark spanning six common English dialects.<n>We design a general encoder-based mitigation strategy for multimodal generative models.
  arXiv  Detail & Related papers  (2025-10-16T17:56:55Z)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models [50.43793764203352]
  We introduce MDK12-Bench, a multi-disciplinary benchmark assessing the reasoning capabilities of MLLMs via real-world K-12 examinations.<n>Our benchmark comprises 140K reasoning instances across diverse difficulty levels from primary school to 12th grade.<n>It features 6,827 instance-level knowledge point annotations based on a well-organized knowledge structure, detailed answer explanations, difficulty labels and cross-year partitions.
  arXiv  Detail & Related papers  (2025-04-08T08:06:53Z)
• Comparative Insights from 12 Machine Learning Models in Extracting Economic Ideology from Political Text [0.0]
  This study conducts a systematic assessment of the capabilities of 12 machine learning models and model variations in detecting economic ideology.<n>The analysis assesses the performance of several generative, fine-tuned, and zero-shot models at the granular and aggregate levels.
  arXiv  Detail & Related papers  (2025-01-16T18:06:22Z)
• Eureka: Evaluating and Understanding Large Foundation Models [23.020996995362104]
  We present Eureka, an open-source framework for standardizing evaluations of large foundation models beyond single-score reporting and rankings. We conduct an analysis of 12 state-of-the-art models, providing in-depth insights into failure understanding and model comparison.
  arXiv  Detail & Related papers  (2024-09-13T18:01:49Z)
• A Comprehensive Evaluation and Analysis Study for Chinese Spelling Check [53.152011258252315]
  We show that using phonetic and graphic information reasonably is effective for Chinese Spelling Check. Models are sensitive to the error distribution of the test set, which reflects the shortcomings of models. The commonly used benchmark, SIGHAN, can not reliably evaluate models' performance.
  arXiv  Detail & Related papers  (2023-07-25T17:02:38Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)
• ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter [19.830089364830066]
  ArtGPT-4 is a large vision-language model tailored to address the limitations of existing models in artistic comprehension. It can render images with an artistic-understanding and convey the emotions they inspire, mirroring human interpretation.
  arXiv  Detail & Related papers  (2023-05-12T14:04:30Z)
• Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification [58.720142291102135]
  This case study investigates the task of job classification in a real-world setting. The goal is to determine whether an English-language job posting is appropriate for a graduate or entry-level position.
  arXiv  Detail & Related papers  (2023-03-13T14:09:53Z)
• Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning [104.58874584354787]
  In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models.
  arXiv  Detail & Related papers  (2023-01-27T18:59:01Z)
• Labeling Explicit Discourse Relations using Pre-trained Language Models [0.0]
  State-of-the-art models achieve slightly above 45% of F-score by using hand-crafted features. We find that the pre-trained language models, when finetuned, are powerful enough to replace the linguistic features. This is the first time when a model outperforms the knowledge intensive models without employing any linguistic features.
  arXiv  Detail & Related papers  (2020-06-21T17:18:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.