Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images

• URL: http://arxiv.org/abs/2604.07338v1
• Date: Wed, 08 Apr 2026 17:53:26 GMT
• Title: Appear2Meaning: A Cross-Cultural Benchmark for Structured Cultural Metadata Inference from Images
• Abstract summary: We introduce a multi-category, cross-cultural benchmark for this task and evaluate vision-language models (VLMs)<n>To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions.<n>Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions.
• Score: 28.09239790313765
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in vision-language models (VLMs) have improved image captioning for cultural heritage. However, inferring structured cultural metadata (e.g., creator, origin, period) from visual input remains underexplored. We introduce a multi-category, cross-cultural benchmark for this task and evaluate VLMs using an LLM-as-Judge framework that measures semantic alignment with reference annotations. To assess cultural reasoning, we report exact-match, partial-match, and attribute-level accuracy across cultural regions. Results show that models capture fragmented signals and exhibit substantial performance variation across cultures and metadata types, leading to inconsistent and weakly grounded predictions. These findings highlight the limitations of current VLMs in structured cultural metadata inference beyond visual perception.

Related papers

• Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluation [2.3757872686260506]
  Vision-language models (VLMs) have achieved strong performance on tasks such as image captioning, visual question answering, and image-to-text generation.<n>They are predominantly trained on English-centric data, which limits their ability to handle culturally grounded visual understanding.<n>We introduce PoVisLE, a monocultural vision-language benchmark for Polish designed to evaluate culturally grounded multimodal understanding.
  arXiv  Detail & Related papers  (2026-08-07T21:01:14Z)
• The Culture Funnel: You Can't Align What isn't in the Data [13.124454342678055]
  We show explicit cultural signals decline sharply during post-training, while geographically concentrated, task-specialized data dominates.<n>Our tags improve downstream cultural benchmark performance, demonstrating that advances require shifting focus in training data pipelines.
  arXiv  Detail & Related papers  (2026-06-11T18:21:10Z)
• CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs [13.267566050457974]
  Cross-cultural knowledge insertion focuses on adapting models to specific cultural contexts while preserving original behavior in other cultures.<n> benchmark includes 9,800 image-grounded cases covering 49 culturally relevant visual scenarios across English, Chinese, and Arabic language-culture groups.<n>Memory-Conditioned Knowledge Insertion (MCKI) retrieves relevant cultural knowledge from an external memory using frozen MLLM representations.
  arXiv  Detail & Related papers  (2026-05-07T12:26:11Z)
• CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs [24.598338950728234]
  Large language models (LLMs) are increasingly deployed in culturally diverse environments.<n>Existing methods focus on de-contextualized correctness or forced-choice judgments.<n>We introduce a set of benchmarks that present models with realistic situational contexts.
  arXiv  Detail & Related papers  (2025-11-15T03:39:13Z)
• Culturally-Aware Conversations: A Framework & Benchmark for LLMs [8.314136556868563]
  Existing benchmarks that measure cultural adaptation in LLMs are misaligned with the actual challenges these models face when interacting with users from diverse cultural backgrounds.<n>Grounded in sociocultural theory, our framework formalizes how linguistic style is shaped by situational, relational, and cultural context.<n>We construct a benchmark dataset based on this framework, annotated by culturally diverse raters, and propose a new set of desiderata for cross-cultural evaluation in NLP.
  arXiv  Detail & Related papers  (2025-10-13T16:06:14Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMs [62.9861554207279]
  Adapting cultural values in Large Language Models (LLMs) presents significant challenges.<n>Prior work primarily aligns LLMs with different cultural values using World Values Survey (WVS) data.<n>We investigate WVS-based training for cultural value adaptation and find that relying solely on survey data cane cultural norms and interfere with factual knowledge.
  arXiv  Detail & Related papers  (2025-05-22T09:00:01Z)
• RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding [79.44246283490665]
  We introduce RAVENEA, a new benchmark designed to advance visual culture understanding through retrieval.<n>RAVENEA focuses on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC)<n>We train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art vision-language models.
  arXiv  Detail & Related papers  (2025-05-20T14:57:16Z)
• Cultural Learning-Based Culture Adaptation of Language Models [70.1063219524999]
  Adapting large language models (LLMs) to diverse cultural values is a challenging task.<n>We present CLCA, a novel framework for enhancing LLM alignment with cultural values based on cultural learning.
  arXiv  Detail & Related papers  (2025-04-03T18:16:26Z)
• CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries [63.00147630084146]
  Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding.<n>CultureVerse is a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types.<n>We propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding.
  arXiv  Detail & Related papers  (2025-01-02T14:42:37Z)
• CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts [45.77570690529597]
  We introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts.<n>Our evaluation of several state-of-the-art open Vision and Language models shows large performance disparities between culture-specific and common concepts.<n>Experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions.
  arXiv  Detail & Related papers  (2024-10-20T17:31:19Z)
• On the Cultural Gap in Text-to-Image Generation [75.69755281031951]
  One challenge in text-to-image (T2I) generation is the inadvertent reflection of culture gaps present in the training data. There is no benchmark to systematically evaluate a T2I model's ability to generate cross-cultural images. We propose a Challenging Cross-Cultural (C3) benchmark with comprehensive evaluation criteria, which can assess how well-suited a model is to a target culture.
  arXiv  Detail & Related papers  (2023-07-06T13:17:55Z)
• Benchmarking Machine Translation with Cultural Awareness [50.183458829028226]
  Translating culture-related content is vital for effective cross-cultural communication. Many culture-specific items (CSIs) often lack viable translations across languages. This difficulty hinders the analysis of cultural awareness of machine translation systems.
  arXiv  Detail & Related papers  (2023-05-23T17:56:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.