Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theory

• URL: http://arxiv.org/abs/2605.15990v1
• Date: Fri, 15 May 2026 14:21:52 GMT
• Title: Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theory
• Abstract summary: We propose a three-level taxonomy of AI-relevant cultural capabilities.<n>We position this taxonomy as a practical tool for improving the validity and interpretability of AI evaluation in real-world, multicultural settings.
• Score: 6.132870531764035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tremendous efforts have been put into evaluating the inclusivity and effectiveness of AI systems across cultures. However, the cultural capabilities considered in much of the literature remain vaguely defined, are referred to using interchangeable terminology, and are typically limited to recalling accurate information about various demographics, regions, and nationalities. To address this construct ambiguity, we draw from Intercultural Communication scholarship and propose a three-level taxonomy of AI-relevant cultural capabilities: Cultural Awareness answers "Does the model know?", Cultural Sensitivity answers "How does it frame its knowledge?", and Cultural Competence answers "Can it adapt as the interaction evolves?". Beyond conceptual clarification, we position this taxonomy as a practical tool for improving the validity and interpretability of AI evaluation in real-world, multicultural settings. Without such construct clarity, evaluation results risk overstating model capabilities and may lead to inappropriate deployment decisions in culturally sensitive contexts.

Related papers

• CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs [59.09971492475217]
  We introduce CultureForest, a benchmark for textitCultural Norm Grounded Reasoning<n>CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions.<n>Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings.
  arXiv  Detail & Related papers  (2026-06-01T08:25:12Z)
• CALM: Culturally Self-Aware Language Models [22.309957211042597]
  CALM is a novel framework designed to endow language models with cultural self-awareness.<n>It disentangles task semantics from explicit cultural concepts and latent cultural signals.<n>The resulting unified representation is fused with the model's original knowledge to construct a culturally grounded internal identity state.
  arXiv  Detail & Related papers  (2026-01-07T00:28:33Z)
• 'Too much alignment; not enough culture': Re-balancing cultural alignment practices in LLMs [0.0]
  This paper argues for a shift towards integrating qualitative approaches into AI alignment practices.<n> Drawing inspiration from Clifford Geertz's concept of "thick description," we propose that AI systems must produce outputs that reflect deeper cultural meanings.
  arXiv  Detail & Related papers  (2025-09-30T12:22:53Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• Culture is Everywhere: A Call for Intentionally Cultural Evaluation [36.20861746863831]
  We argue for textbfintentionally cultural evaluation: an approach that systematically examines the cultural assumptions embedded in all aspects of evaluation.<n>We discuss implications and future directions for moving beyond current benchmarking practices.
  arXiv  Detail & Related papers  (2025-09-01T09:39:21Z)
• From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test [50.51344198689069]
  We extend the human-centered word association test (WAT) to assess the alignment of large language models with cross-cultural cognition.<n>To address culture preference, we propose CultureSteer, an innovative approach by embedding cultural-specific semantic associations directly within the model's internal representation space.
  arXiv  Detail & Related papers  (2025-05-24T07:05:10Z)
• CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts [45.77570690529597]
  We introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts.<n>Our evaluation of several state-of-the-art open Vision and Language models shows large performance disparities between culture-specific and common concepts.<n>Experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions.
  arXiv  Detail & Related papers  (2024-10-20T17:31:19Z)
• Extrinsic Evaluation of Cultural Competence in Large Language Models [53.626808086522985]
  We focus on extrinsic evaluation of cultural competence in two text generation tasks. We evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. We find weak correlations between text similarity of outputs for different countries and the cultural values of these countries.
  arXiv  Detail & Related papers  (2024-06-17T14:03:27Z)
• Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking [48.21982147529661]
  This paper introduces a novel approach for massively multicultural knowledge acquisition. Our method strategically navigates from densely informative Wikipedia documents on cultural topics to an extensive network of linked pages. Our work marks an important step towards deeper understanding and bridging the gaps of cultural disparities in AI.
  arXiv  Detail & Related papers  (2024-02-14T18:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.