AI and Cultural Context: An Empirical Investigation of Large Language Models' Performance on Chinese Social Work Professional Standards

• URL: http://arxiv.org/abs/2412.14971v1
• Date: Thu, 19 Dec 2024 15:48:27 GMT
• Title: AI and Cultural Context: An Empirical Investigation of Large Language Models' Performance on Chinese Social Work Professional Standards
• Abstract summary: This study examines how well leading Chinese and Western large language models understand and apply Chinese social work principles.<n>We test whether the cultural context in the developing country influences model reasoning and accuracy.
• Score: 2.0275382001913487
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Objective: This study examines how well leading Chinese and Western large language models understand and apply Chinese social work principles, focusing on their foundational knowledge within a non-Western professional setting. We test whether the cultural context in the developing country influences model reasoning and accuracy. Method: Using a published self-study version of the Chinese National Social Work Examination (160 questions) covering jurisprudence and applied knowledge, we administered three testing conditions to eight cloud-based large language models - four Chinese and four Western. We examined their responses following official guidelines and evaluated their explanations' reasoning quality. Results: Seven models exceeded the 60-point passing threshold in both sections. Chinese models performed better in jurisprudence (median = 77.0 vs. 70.3) but slightly lower in applied knowledge (median = 65.5 vs. 67.0). Both groups showed cultural biases, particularly regarding gender equality and family dynamics. Models demonstrated strong professional terminology knowledge but struggled with culturally specific interventions. Valid reasoning in incorrect answers ranged from 16.4% to 45.0%. Conclusions: While both Chinese and Western models show foundational knowledge of Chinese social work principles, technical language proficiency does not ensure cultural competence. Chinese models demonstrate advantages in regulatory content, yet both Chinese and Western models struggle with culturally nuanced practice scenarios. These findings contribute to informing responsible AI integration into cross-cultural social work practice.

Related papers

• CCBENCH: Assessing LLM Cultural Competence via Implicitly Signaled Norms using Health Queries [42.15271207948674]
  We introduce CCBENCH, a framework for evaluating cultural competency in large language models (LLMs)<n>As a case study on health, we create CCBENCH-Health, which includes 60 theoretically grounded personas exhibiting varied norm-adherence states across six cultures.<n> Benchmarking five leading models reveals that even the best achieve culturally appropriate responses only 20-30% of the time.
  arXiv  Detail & Related papers  (2026-06-08T22:52:34Z)
• CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs [59.09971492475217]
  We introduce CultureForest, a benchmark for textitCultural Norm Grounded Reasoning<n>CultureForest comprises 5,378 examples across 8 domains and 53 countries/regions.<n>Extensive experiments reveal that even top-tier models degrade substantially in open-ended settings.
  arXiv  Detail & Related papers  (2026-06-01T08:25:12Z)
• Pragmatics Meets Culture: Culturally-adapted Artwork Description Generation and Evaluation [26.09933226566947]
  We introduce the task of culturally-adapted art description generation, where models describe artworks for audiences from different cultural groups who vary in their familiarity with the cultural symbols and narratives embedded in the artwork.<n>We find that base models are only marginally adequate for this task, but, through a pragmatic speaker model, we can improve simulated listener comprehension by up to 8.2%.<n>A human study confirms that the model with higher pragmatic competence is rated as more helpful for comprehension by 8.0%.
  arXiv  Detail & Related papers  (2026-04-02T22:16:07Z)
• The performances of the Chinese and U.S. Large Language Models on the Topic of Chinese Culture [11.591457038182838]
  This study adopts a direct-questioning paradigm to evaluate models such as GPT-5.1, DeepSeek-V3.2, Qwen3-Max, and Gemini2.5Pro.<n>We assess their understanding of traditional Chinese culture, including history, literature, poetry, and related domains.
  arXiv  Detail & Related papers  (2026-01-06T09:03:01Z)
• The American Ghost in the Machine: How language models align culturally and the effects of cultural prompting [0.0]
  We use the VSM13 International Survey and Hofstede's cultural dimensions to identify the cultural alignment of popular Large Language Models (LLMs)<n>We then use cultural prompting to test the adaptability of these models to other cultures, namely China, France, India, Iran, Japan, and the United States.<n>We find that the majority of the eight LLMs tested favor the United States when the culture is not specified, with varying results when prompted for other cultures.
  arXiv  Detail & Related papers  (2025-12-13T23:11:41Z)
• Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge [68.6805229085352]
  Most multilingual question-answering benchmarks do not factor in regional diversity in the information they capture.<n>XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages.<n>We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics.
  arXiv  Detail & Related papers  (2025-11-01T18:41:34Z)
• MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation [91.22008265721952]
  MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned benchmark covering 8 Asian countries and 10 languages.<n>This is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech.<n>We propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity.
  arXiv  Detail & Related papers  (2025-10-07T14:12:12Z)
• Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World [68.19795061447044]
  This paper investigates cross-cultural transfer of commonsense reasoning in the Arab world.<n>Using a culturally grounded commonsense reasoning dataset covering 13 Arab countries, we evaluate lightweight alignment methods.<n>Our results show that merely 12 culture-specific examples from one country can improve performance in others by 10% on average.
  arXiv  Detail & Related papers  (2025-09-23T17:24:14Z)
• MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs [25.128936333806678]
  Large language models exhibit cultural biases and limited cross-cultural understanding capabilities.<n>We propose MCEval, a novel multilingual evaluation framework that employs dynamic cultural question construction.
  arXiv  Detail & Related papers  (2025-07-13T16:24:35Z)
• Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition [0.0]
  We introduce Nunchi-Bench, a benchmark designed to evaluate large language models' cultural understanding.<n>The benchmark consists of 247 questions spanning 31 topics, assessing factual knowledge, culturally appropriate advice, and situational interpretation.<n>We evaluate multilingual LLMs in both Korean and English to analyze their ability to reason about Korean cultural contexts.
  arXiv  Detail & Related papers  (2025-07-05T11:52:09Z)
• VideoVista-CulturalLingo: 360$^\circ$ Horizons-Bridging Cultures, Languages, and Domains in Video Comprehension [66.03062468036507]
  We present VideoVista-CulturalLingo, the first video evaluation benchmark designed to bridge cultural, linguistic, and domain divide in video comprehension. VideoVista-CulturalLingo contains 1,389 videos and 3,134 QA pairs, and we have evaluated 24 recent open-source or proprietary video large models.
  arXiv  Detail & Related papers  (2025-04-23T13:47:30Z)
• DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers [17.355452637877402]
  We conduct the first cultural evaluation study for the mid-resource language of Danish, in which native speakers prompt different models to solve tasks requiring cultural awareness. Our analysis of the resulting 1,038 interactions from 63 demographically diverse participants highlights open challenges to cultural adaptation.
  arXiv  Detail & Related papers  (2025-04-03T08:52:42Z)
• CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries [63.00147630084146]
  Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding. CultureVerse is a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types. We propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding.
  arXiv  Detail & Related papers  (2025-01-02T14:42:37Z)
• Risks of Cultural Erasure in Large Language Models [4.613949381428196]
  We argue for the need of metricizable evaluations of language technologies that interrogate and account for historical power inequities. We probe representations that a language model produces about different places around the world when asked to describe these contexts. We analyze the cultures represented in the travel recommendations produced by a set of language model applications.
  arXiv  Detail & Related papers  (2025-01-02T04:57:50Z)
• Whose Morality Do They Speak? Unraveling Cultural Bias in Multilingual Language Models [0.0]
  Large language models (LLMs) have become integral tools in diverse domains, yet their moral reasoning capabilities remain underexplored. This study investigates whether multilingual LLMs, such as GPT-3.5-Turbo, reflect culturally specific moral values or impose dominant moral norms. Using the updated Moral Foundations Questionnaire (MFQ-2) in eight languages, the study analyzes the models' adherence to six core moral foundations.
  arXiv  Detail & Related papers  (2024-12-25T10:17:15Z)
• CulturalBench: a Robust, Diverse and Challenging Benchmark on Measuring the (Lack of) Cultural Knowledge of LLMs [75.82306181299153]
  We introduce CulturalBench: a set of 1,227 human-written and human-verified questions for assessing cultural knowledge. We evaluate models on two setups: CulturalBench-Easy and CulturalBench-Hard which share the same questions but asked differently. Compared to human performance (92.6% accuracy), CulturalBench-Hard is more challenging for frontier LLMs with the best performing model (GPT-4o) at only 61.5% and the worst (Llama3-8b) at 21.4%.
  arXiv  Detail & Related papers  (2024-10-03T17:04:31Z)
• Edu-Values: Towards Evaluating the Chinese Education Values of Large Language Models [9.761584874383873]
  We present Edu-Values, the first Chinese education values evaluation benchmark designed to measure large language models' alignment ability. We meticulously design and compile 1,418 questions, including multiple-choice, multi-modal question answering, subjective analysis, adversarial prompts, and questions on traditional Chinese culture. Due to differences in educational culture, Chinese LLMs significantly outperform English LLMs, with Qwen 2 ranking the first with a score of 81.37.
  arXiv  Detail & Related papers  (2024-09-19T13:02:54Z)
• Extrinsic Evaluation of Cultural Competence in Large Language Models [53.626808086522985]
  We focus on extrinsic evaluation of cultural competence in two text generation tasks. We evaluate model outputs when an explicit cue of culture, specifically nationality, is perturbed in the prompts. We find weak correlations between text similarity of outputs for different countries and the cultural values of these countries.
  arXiv  Detail & Related papers  (2024-06-17T14:03:27Z)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models [63.452948673344395]
  This paper introduces CulturePark, an LLM-powered multi-agent communication framework for cultural data collection. It generates high-quality cross-cultural dialogues encapsulating human beliefs, norms, and customs. We evaluate these models across three downstream tasks: content moderation, cultural alignment, and cultural education.
  arXiv  Detail & Related papers  (2024-05-24T01:49:02Z)
• Cultural Bias and Cultural Alignment of Large Language Models [0.9374652839580183]
  We conduct a disaggregated evaluation of cultural bias for five widely used large language models. All models exhibit cultural values resembling English-speaking and Protestant European countries. We suggest using cultural prompting and ongoing evaluation to reduce cultural bias in the output of generative AI.
  arXiv  Detail & Related papers  (2023-11-23T16:45:56Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)
• Cultural Alignment in Large Language Models: An Explanatory Analysis Based on Hofstede's Cultural Dimensions [10.415002561977655]
  This research proposes a Cultural Alignment Test (Hoftede's CAT) to quantify cultural alignment using Hofstede's cultural dimension framework. We quantitatively evaluate large language models (LLMs) against the cultural dimensions of regions like the United States, China, and Arab countries. Our results quantify the cultural alignment of LLMs and reveal the difference between LLMs in explanatory cultural dimensions.
  arXiv  Detail & Related papers  (2023-08-25T14:50:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.