CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages

• URL: http://arxiv.org/abs/2607.21016v1
• Date: Thu, 23 Jul 2026 08:02:44 GMT
• Title: CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languages
• Abstract summary: We introduce CultureTalk-ID, the first dialogue-based benchmark for cultural commonsense in Indonesian and its local languages.<n>CultureTalk-ID comprises 4,496 culturally grounded dialogues across 11 languages and 13 culturally salient topics, curated through a multi-stage human pipeline with native speakers to ensure authenticity.
• Score: 11.10195396318637
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Culture is lived through conversation, yet existing Indonesian cultural commonsense benchmarks evaluate LLMs on short and isolated prompts, stripping away the dialogic context in which cultural nuances actually surface. We introduce CultureTalk-ID, the first dialogue-based benchmark for cultural commonsense in Indonesian and its local languages, comprising 4,496 culturally grounded dialogues across 11 languages and 13 culturally salient topics, curated through a multi-stage human pipeline with native speakers to ensure authenticity. CultureTalk-ID introduces three complementary tasks, namely dialogue-based multiple-choice cultural commonsense reasoning, culturally faithful machine translation, and language steering, which jointly probe whether LLMs can understand, transfer, and generate culturally grounded language.

Related papers

• MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation [91.22008265721952]
  MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned benchmark covering 8 Asian countries and 10 languages.<n>This is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech.<n>We propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity.
  arXiv  Detail & Related papers  (2025-10-07T14:12:12Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs [57.653830744706305]
  CultureScope is the most comprehensive evaluation framework to date for assessing cultural understanding in large language models.<n>Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification.<n> Experimental results demonstrate that our method can effectively evaluate cultural understanding.
  arXiv  Detail & Related papers  (2025-09-19T17:47:48Z)
• MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs [25.128936333806678]
  Large language models exhibit cultural biases and limited cross-cultural understanding capabilities.<n>We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction.
  arXiv  Detail & Related papers  (2025-07-13T16:24:35Z)
• MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs [37.98920430188422]
  MAKIEval is an automatic multilingual framework for evaluating cultural awareness in large language models.<n>It automatically identifies cultural entities in model outputs and links them to structured knowledge.<n>We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems.
  arXiv  Detail & Related papers  (2025-05-27T19:29:40Z)
• DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers [17.355452637877402]
  We conduct the first cultural evaluation study for the mid-resource language of Danish, in which native speakers prompt different models to solve tasks requiring cultural awareness.<n>Our analysis of the resulting 1,038 interactions from 63 demographically diverse participants highlights open challenges to cultural adaptation.
  arXiv  Detail & Related papers  (2025-04-03T08:52:42Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)
• Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense [98.09670425244462]
  Large language models (LLMs) have demonstrated substantial commonsense understanding. This paper examines the capabilities and limitations of several state-of-the-art LLMs in the context of cultural commonsense tasks.
  arXiv  Detail & Related papers  (2024-05-07T20:28:34Z)
• CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting [73.94059188347582]
  We uncover culture perceptions of three SOTA models on 110 countries and regions on 8 culture-related topics through culture-conditioned generations. We discover that culture-conditioned generation consist of linguistic "markers" that distinguish marginalized cultures apart from default cultures.
  arXiv  Detail & Related papers  (2024-04-16T00:50:43Z)
• Cultural Commonsense Knowledge for Intercultural Dialogues [31.079990829088857]
  This paper presents MANGO, a methodology for distilling high-accuracy, high-recall assertions of cultural knowledge. Running the MANGO method with GPT-3.5 as underlying LLM yields 167K high-accuracy assertions for 30K concepts and 11K cultures. We find that adding knowledge from MANGO improves the overall quality, specificity, and cultural sensitivity of dialogue responses.
  arXiv  Detail & Related papers  (2024-02-16T13:46:38Z)
• Bridging Cultural Nuances in Dialogue Agents through Cultural Value Surveys [20.82269206759988]
  cuDialog is a first-of-its-kind benchmark for dialogue generation with a cultural lens. We develop baseline models capable of extracting cultural attributes from dialogue exchanges. We propose to incorporate cultural dimensions with dialogue encoding features.
  arXiv  Detail & Related papers  (2024-01-18T19:42:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.