GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking

• URL: http://arxiv.org/abs/2502.13766v1
• Date: Wed, 19 Feb 2025 14:27:40 GMT
• Title: GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking
• Authors: Florian Schneider, Carolin Holtermann, Chris Biemann, Anne Lauscher,
• Abstract summary: We introduce GIMMICK, an extensive benchmark designed to assess a broad spectrum of cultural knowledge across 144 countries. GIMMICK comprises six tasks built upon three new datasets that span 728 unique cultural events or facets. We examine (1) regional cultural biases, (2) the influence of model size, (3) input modalities, and (4) external cues.
• Score: 29.664707739055068
• License:
• Abstract: Large Vision-Language Models (LVLMs) have recently gained attention due to their distinctive performance and broad applicability. While it has been previously shown that their efficacy in usage scenarios involving non-Western contexts falls short, existing studies are limited in scope, covering just a narrow range of cultures, focusing exclusively on a small number of cultural aspects, or evaluating a limited selection of models on a single task only. Towards globally inclusive LVLM research, we introduce GIMMICK, an extensive multimodal benchmark designed to assess a broad spectrum of cultural knowledge across 144 countries representing six global macro-regions. GIMMICK comprises six tasks built upon three new datasets that span 728 unique cultural events or facets on which we evaluated 20 LVLMs and 11 LLMs, including five proprietary and 26 open-weight models of all sizes. We systematically examine (1) regional cultural biases, (2) the influence of model size, (3) input modalities, and (4) external cues. Our analyses reveal strong biases toward Western cultures across models and tasks and highlight strong correlations between model size and performance, as well as the effectiveness of multimodal input and external geographic cues. We further find that models have more knowledge of tangible than intangible aspects (e.g., food vs. rituals) and that they excel in recognizing broad cultural origins but struggle with a more nuanced understanding.

Related papers

• CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries [63.00147630084146]
  Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding. CultureVerse is a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types. We propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding.
  arXiv  Detail & Related papers  (2025-01-02T14:42:37Z)
• Risks of Cultural Erasure in Large Language Models [4.613949381428196]
  We argue for the need of metricizable evaluations of language technologies that interrogate and account for historical power inequities. We probe representations that a language model produces about different places around the world when asked to describe these contexts. We analyze the cultures represented in the travel recommendations produced by a set of language model applications.
  arXiv  Detail & Related papers  (2025-01-02T04:57:50Z)
• All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages [73.93600813999306]
  ALM-bench is the largest and most comprehensive effort to date for evaluating LMMs across 100 languages. It challenges existing models by testing their ability to understand and reason about culturally diverse images paired with text in various languages. The benchmark offers a robust and nuanced evaluation framework featuring various question formats, including true/false, multiple choice, and open-ended questions.
  arXiv  Detail & Related papers  (2024-11-25T15:44:42Z)
• CulturalBench: a Robust, Diverse and Challenging Benchmark on Measuring the (Lack of) Cultural Knowledge of LLMs [75.82306181299153]
  We introduce CulturalBench: a set of 1,227 human-written and human-verified questions for assessing cultural knowledge. We evaluate models on two setups: CulturalBench-Easy and CulturalBench-Hard which share the same questions but asked differently. Compared to human performance (92.6% accuracy), CulturalBench-Hard is more challenging for frontier LLMs with the best performing model (GPT-4o) at only 61.5% and the worst (Llama3-8b) at 21.4%.
  arXiv  Detail & Related papers  (2024-10-03T17:04:31Z)
• Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models [22.92083941222383]
  We introduce DalleStreet, a large-scale dataset generated by DALL-E 3 and validated by humans. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models. Our findings reveal a nuanced picture of the cultural competence of LMMs, highlighting the need to develop culture-aware systems.
  arXiv  Detail & Related papers  (2024-07-02T08:55:41Z)
• From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models [10.121734731147376]
  Vision-language models' performance remains suboptimal on images from non-western cultures. Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures. We introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding.
  arXiv  Detail & Related papers  (2024-06-28T23:28:28Z)
• Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
  Large language models (LLMs) have demonstrated substantial commonsense understanding. This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
  arXiv  Detail & Related papers  (2024-05-07T20:28:34Z)
• Towards Measuring and Modeling "Culture" in LLMs: A Survey [21.94407169332458]
  We present a survey of more than 90 recent papers that aim to study cultural representation and inclusion in large language models (LLMs) We observe that none of the studies explicitly define "culture" We call these aspects the proxies of culture, and organize them across two dimensions of demographic and semantic proxies.
  arXiv  Detail & Related papers  (2024-03-05T08:29:36Z)
• Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking [48.21982147529661]
  This paper introduces a novel approach for massively multicultural knowledge acquisition. Our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI.
  arXiv  Detail & Related papers  (2024-02-14T18:16:54Z)
• CDEval: A Benchmark for Measuring the Cultural Dimensions of Large Language Models [41.885600036131045]
  CDEval is a benchmark aimed at evaluating the cultural dimensions of Large Language Models. It is constructed by incorporating both GPT-4's automated generation and human verification, covering six cultural dimensions across seven domains.
  arXiv  Detail & Related papers  (2023-11-28T02:01:25Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.