Culturally-Aware Conversations: A Framework & Benchmark for LLMs

• URL: http://arxiv.org/abs/2510.11563v1
• Date: Mon, 13 Oct 2025 16:06:14 GMT
• Title: Culturally-Aware Conversations: A Framework & Benchmark for LLMs
• Authors: Shreya Havaldar, Sunny Rai, Young-Min Cho, Lyle Ungar,
• Abstract summary: Existing benchmarks that measure cultural adaptation in LLMs are misaligned with the actual challenges these models face when interacting with users from diverse cultural backgrounds.<n>Grounded in sociocultural theory, our framework formalizes how linguistic style is shaped by situational, relational, and cultural context.<n>We construct a benchmark dataset based on this framework, annotated by culturally diverse raters, and propose a new set of desiderata for cross-cultural evaluation in NLP.
• Score: 8.314136556868563
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Existing benchmarks that measure cultural adaptation in LLMs are misaligned with the actual challenges these models face when interacting with users from diverse cultural backgrounds. In this work, we introduce the first framework and benchmark designed to evaluate LLMs in realistic, multicultural conversational settings. Grounded in sociocultural theory, our framework formalizes how linguistic style - a key element of cultural communication - is shaped by situational, relational, and cultural context. We construct a benchmark dataset based on this framework, annotated by culturally diverse raters, and propose a new set of desiderata for cross-cultural evaluation in NLP: conversational framing, stylistic sensitivity, and subjective correctness. We evaluate today's top LLMs on our benchmark and show that these models struggle with cultural adaptation in a conversational setting.

Related papers

• Understanding Cultural Alignment in Multilingual LLMs via Natural Debate Statements [20.067090212539217]
  This work investigates the sociocultural values learned by large language models (LLMs)<n>We introduce a novel open-access dataset, Sociocultural Statements, constructed from natural debate statements using a multi-step methodology.<n>The dataset is synthetically labeled to enable the quantization of sociocultural norms and beliefs that LLMs exhibit in their responses to these statements.
  arXiv  Detail & Related papers  (2026-02-13T12:27:54Z)
• Do Large Language Models Truly Understand Cross-cultural Differences? [53.481048019144644]
  We develop a scenario-based benchmark to evaluate large language models' cross-cultural understanding and reasoning.<n>Grounded in cultural theory, we categorize cross-cultural capabilities into nine dimensions.<n>The dataset supports continuous expansion, and experiments confirm its transferability to other languages.
  arXiv  Detail & Related papers  (2025-12-08T01:21:58Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test [50.51344198689069]
  We extend the human-centered word association test (WAT) to assess the alignment of large language models with cross-cultural cognition.<n>To address culture preference, we propose CultureSteer, an innovative approach by embedding cultural-specific semantic associations directly within the model's internal representation space.
  arXiv  Detail & Related papers  (2025-05-24T07:05:10Z)
• From Surveys to Narratives: Rethinking Cultural Value Adaptation in LLMs [62.9861554207279]
  Adapting cultural values in Large Language Models (LLMs) presents significant challenges.<n>Prior work primarily aligns LLMs with different cultural values using World Values Survey (WVS) data.<n>We investigate WVS-based training for cultural value adaptation and find that relying solely on survey data cane cultural norms and interfere with factual knowledge.
  arXiv  Detail & Related papers  (2025-05-22T09:00:01Z)
• Cultural Learning-Based Culture Adaptation of Language Models [70.1063219524999]
  Adapting large language models (LLMs) to diverse cultural values is a challenging task.<n>We present CLCA, a novel framework for enhancing LLM alignment with cultural values based on cultural learning.
  arXiv  Detail & Related papers  (2025-04-03T18:16:26Z)
• Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning [20.24633389489255]
  We present a simple and inexpensive method that uses a combination of in-context learning (ICL) and human survey data.<n>We show that our method could prove useful in test languages other than English and can improve alignment to the cultural values that correspond to a range of culturally diverse countries.
  arXiv  Detail & Related papers  (2024-08-29T12:18:04Z)
• Translating Across Cultures: LLMs for Intralingual Cultural Adaptation [12.5954253354303]
  We define the task of cultural adaptation and create an evaluation framework to evaluate the performance of modern LLMs. We analyze possible issues with automatic adaptation. We hope that this paper will offer more insight into the cultural understanding of LLMs and their creativity in cross-cultural scenarios.
  arXiv  Detail & Related papers  (2024-06-20T17:06:58Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.