CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

• URL: http://arxiv.org/abs/2608.28405v1
• Date: Fri, 28 Aug 2026 14:56:46 GMT
• Title: CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia
• Abstract summary: We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue.<n>The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues.<n>Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment.
• Score: 72.20656906184539
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common use case: users seeking practical help over multiple turns in culturally grounded scenarios. We introduce CultureConverse, a scalable, multilingual simulation and evaluation harness for culturally grounded assistant dialogue that covers 10 East and Southeast Asian regions, 58 subgroup identities, and 7 domains. Each simulated and evaluated episode produces a scored interaction where the assistant assists the user and infers cultural constraints from partial information. The resulting CultureConverse-DS dataset contains 14,610 benchmark (evaluation) episodes and 274,295 oracle-guided (gold-mode) dialogues. In our benchmark evaluation of 18 models, GPT-5 mini achieves the highest assistance quality. Human annotation experiments suggest that our evaluation framework is a sufficient proxy for human judgment. Performance gains from fine-tuning on 27,860 high-quality CultureConverse-DS samples improve in-domain assistance and transfer out-of-domain to cultural MCQ and safety classification benchmarks. We release the harness, both splits, and judge prompts to support interactive evaluation of cultural competency.

Related papers

• CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries [42.15271207948674]
  We introduce CCBENCH, a framework for evaluating cultural competency in large language models (LLMs)<n>As a case study on health, we create CCBENCH-Health, which includes 60 theoretically grounded personas exhibiting varied norm-adherence states across six cultures.<n> Benchmarking five leading models reveals that even the best achieve culturally appropriate responses only 20-30% of the time.
  arXiv  Detail & Related papers  (2026-06-08T22:52:34Z)
• CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs [13.267566050457974]
  Cross-cultural knowledge insertion focuses on adapting models to specific cultural contexts while preserving original behavior in other cultures.<n> benchmark includes 9,800 image-grounded cases covering 49 culturally relevant visual scenarios across English, Chinese, and Arabic language-culture groups.<n>Memory-Conditioned Knowledge Insertion (MCKI) retrieves relevant cultural knowledge from an external memory using frozen MLLM representations.
  arXiv  Detail & Related papers  (2026-05-07T12:26:11Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• Culture is Everywhere: A Call for Intentionally Cultural Evaluation [36.20861746863831]
  We argue for textbfintentionally cultural evaluation: an approach that systematically examines the cultural assumptions embedded in all aspects of evaluation.<n>We discuss implications and future directions for moving beyond current benchmarking practices.
  arXiv  Detail & Related papers  (2025-09-01T09:39:21Z)
• MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs [25.128936333806678]
  Large language models exhibit cultural biases and limited cross-cultural understanding capabilities.<n>We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction.
  arXiv  Detail & Related papers  (2025-07-13T16:24:35Z)
• RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding [79.44246283490665]
  We introduce RAVENEA, a new benchmark designed to advance visual culture understanding through retrieval.<n>RAVENEA focuses on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC)<n>We train and evaluate seven multimodal retrievers for each image query, and measure the downstream impact of retrieval-augmented inputs across fourteen state-of-the-art vision-language models.
  arXiv  Detail & Related papers  (2025-05-20T14:57:16Z)
• CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries [63.00147630084146]
  Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding.<n>CultureVerse is a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types.<n>We propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding.
  arXiv  Detail & Related papers  (2025-01-02T14:42:37Z)
• CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming [75.82306181299153]
  CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge.<n>It covers 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru.<n>We construct CulturalBench using methods inspired by Human-AI Red-Teaming.
  arXiv  Detail & Related papers  (2024-10-03T17:04:31Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.