Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities

• URL: http://arxiv.org/abs/2608.29209v1
• Date: Sat, 29 Aug 2026 11:45:39 GMT
• Title: Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities
• Abstract summary: We examine how well AI-generated multimodal stories align with the lived practices, relationships, language, values, and visual expectations of the communities they represent.<n>We conduct a community-grounded mixed-methods evaluation with 19 culture representatives across five African communities.<n>We develop a taxonomy of cultural alignment comprising five broader cultural marker categories and eight recurring mechanisms of misalignment.
• Score: 13.137760017965087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we examine how well AI-generated multimodal stories align with the lived practices, relationships, language, values, and visual expectations of the communities they represent. We conduct a community-grounded mixed-methods evaluation with 19 culture representatives across five African communities, combining quantitative annotations with qualitative focus group discussions. We find that cultural alignment depends not simply on recognizable cultural markers, but on how those markers fit social, linguistic, procedural, and visual context. From these evaluations, we develop a taxonomy of cultural alignment comprising five broader cultural marker categories and eight recurring mechanisms of misalignment. We additionally evaluate five multimodal LLM judges to examine whether automated evaluation can approximate community-grounded judgments at scale. Judge reliability and score calibration vary substantially across communities, with no single judge performing consistently across all five settings. These findings motivate community-calibrated evaluation pipelines in which automated judges are validated against community judgments to determine where they can be trusted and where human review remains necessary.

Related papers

• CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia [72.20656906184539]
  We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue.<n>The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues.<n>Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment.
  arXiv  Detail & Related papers  (2026-08-28T14:56:46Z)
• CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs [59.09971492475217]
  We introduce CultureForest, a benchmark for textitCultural Norm Grounded Reasoning<n>CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions.<n>Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings.
  arXiv  Detail & Related papers  (2026-06-01T08:25:12Z)
• Cultural Authenticity: Comparing LLM Cultural Representations to Native Human Expectations [7.285754268528876]
  We introduce a human-centered framework to evaluate the alignment of Large Language Model generations with local expectations.<n>Our investigation of the alignment between the human-derived Cultural Importance and model-derived Cultural Representations reveals a Western-centric calibration.
  arXiv  Detail & Related papers  (2026-04-03T22:39:15Z)
• Culture is Everywhere: A Call for Intentionally Cultural Evaluation [36.20861746863831]
  We argue for textbfintentionally cultural evaluation: an approach that systematically examines the cultural assumptions embedded in all aspects of evaluation.<n>We discuss implications and future directions for moving beyond current benchmarking practices.
  arXiv  Detail & Related papers  (2025-09-01T09:39:21Z)
• A Game-Theoretic Negotiation Framework for Cross-Cultural Consensus in LLMs [10.655783463895325]
  Large language models (LLMs) exhibit a pronounced WEIRD (Western, Educated, Industrialized, Rich, Democratic) cultural bias.<n>This monocultural perspective may reinforce dominant values and marginalize diverse cultural viewpoints.<n>We introduce a systematic framework designed to boost fair and robust cross-cultural consensus.
  arXiv  Detail & Related papers  (2025-06-16T08:42:39Z)
• CAIRe: Cultural Attribution of Images by Retrieval-Augmented Evaluation [61.130639734982395]
  We introduce CAIRe, a novel evaluation metric that assesses the degree of cultural relevance of an image.<n>Our framework grounds entities and concepts in the image to a knowledge base and uses factual information to give independent graded judgments for each culture label.
  arXiv  Detail & Related papers  (2025-06-10T17:16:23Z)
• The Case for "Thick Evaluations" of Cultural Representation in AI [11.53198252426806]
  Generative AI image models have been increasingly evaluated for their (in)ability to represent non-Western cultures.<n>We argue that these evaluations operate through reductive ideals of representation, neglecting how people define their own representation.<n>We introduce the idea of 'thick evaluations': a more granular, situated, and discursive measurement framework for evaluating representations of social worlds in AI images.
  arXiv  Detail & Related papers  (2025-03-24T19:01:14Z)
• Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMs [7.802103248428407]
  We identify and test three assumptions behind current survey-based evaluation methods.<n>We find a high level of instability across presentation formats, incoherence between evaluated versus held-out cultural dimensions, and erratic behavior under prompt steering.
  arXiv  Detail & Related papers  (2025-03-11T17:59:53Z)
• Rethinking AI Cultural Alignment [1.8434042562191815]
  We show that humans' cultural values must be understood within the context of specific AI systems.<n>We argue that cultural alignment should be reframed as a bidirectional process.
  arXiv  Detail & Related papers  (2025-01-13T23:42:37Z)
• CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries [63.00147630084146]
  Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding.<n>CultureVerse is a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types.<n>We propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding.
  arXiv  Detail & Related papers  (2025-01-02T14:42:37Z)
• Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation [71.59208664920452]
  Cultural biases in multilingual datasets pose significant challenges for their effectiveness as global benchmarks.<n>We show that progress on MMLU depends heavily on learning Western-centric concepts, with 28% of all questions requiring culturally sensitive knowledge.<n>We release Global MMLU, an improved MMLU with evaluation coverage across 42 languages.
  arXiv  Detail & Related papers  (2024-12-04T13:27:09Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.