The performances of the Chinese and U.S. Large Language Models on the Topic of Chinese Culture

• URL: http://arxiv.org/abs/2601.02830v2
• Date: Wed, 07 Jan 2026 03:35:16 GMT
• Title: The performances of the Chinese and U.S. Large Language Models on the Topic of Chinese Culture
• Abstract summary: This study adopts a direct-questioning paradigm to evaluate models such as GPT-5.1, DeepSeek-V3.2, Qwen3-Max, and Gemini2.5Pro.<n>We assess their understanding of traditional Chinese culture, including history, literature, poetry, and related domains.
• Score: 11.591457038182838
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cultural backgrounds shape individuals' perspectives and approaches to problem-solving. Since the emergence of GPT-1 in 2018, large language models (LLMs) have undergone rapid development. To date, the world's ten leading LLM developers are primarily based in China and the United States. To examine whether LLMs released by Chinese and U.S. developers exhibit cultural differences in Chinese-language settings, we evaluate their performance on questions about Chinese culture. This study adopts a direct-questioning paradigm to evaluate models such as GPT-5.1, DeepSeek-V3.2, Qwen3-Max, and Gemini2.5Pro. We assess their understanding of traditional Chinese culture, including history, literature, poetry, and related domains. Comparative analyses between LLMs developed in China and the U.S. indicate that Chinese models generally outperform their U.S. counterparts on these tasks. Among U.S.-developed models, Gemini 2.5Pro and GPT-5.1 achieve relatively higher accuracy. The observed performance differences may potentially arise from variations in training data distribution, localization strategies, and the degree of emphasis on Chinese cultural content during model development.

Related papers

• CDTP: A Large-Scale Chinese Data-Text Pair Dataset for Comprehensive Evaluation of Chinese LLMs [71.01843542502438]
  We present a comprehensive benchmark for evaluating Chinese Large Language Models (CB-ECLLM)<n>CB-ECLLM is based on the newly constructed Chinese Data-Text Pair (CDTP) dataset.<n>CDTP comprises over 7 million aligned text pairs, each consisting of unstructured text coupled with one or more corresponding triples, alongside a total of 15 million triples spanning four critical domains.
  arXiv  Detail & Related papers  (2025-10-07T15:33:52Z)
• TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs [13.069833806549914]
  We propose the Traditional Chinese Culture understanding Benchmark (TCC-Bench) for assessing the understanding of traditional Chinese culture.<n>TCC-Bench comprises culturally rich and visually diverse data, incorporating images from museum artifacts, everyday life scenes, comics, and other culturally significant contexts.<n>We adopt a semi-automated pipeline that utilizes GPT-4o in text-only mode to generate candidate questions, followed by human curation to ensure data quality and avoid potential data leakage.
  arXiv  Detail & Related papers  (2025-05-16T14:10:41Z)
• WenyanGPT: A Large Language Model for Classical Chinese Tasks [36.380841559581945]
  Existing natural language processing models primarily optimize for Modern Chinese, resulting in inadequate performance on Classical Chinese.<n>By continuing pre-training and instruction fine-tuning on the LLaMA3-8B-Chinese model, we construct a large language model, WenyanGPT, which is specifically designed for Classical Chinese tasks.
  arXiv  Detail & Related papers  (2025-04-29T10:19:05Z)
• AI and Cultural Context: An Empirical Investigation of Large Language Models' Performance on Chinese Social Work Professional Standards [2.0275382001913487]
  This study examines how well leading Chinese and Western large language models understand and apply Chinese social work principles.<n>We test whether the cultural context in the developing country influences model reasoning and accuracy.
  arXiv  Detail & Related papers  (2024-12-19T15:48:27Z)
• Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models [24.47838086336772]
  Chinese SimpleQA is the first comprehensive Chinese benchmark to evaluate the factuality ability of language models to answer short questions. We focus on the Chinese language over 6 major topics with 99 diverse subtopics.
  arXiv  Detail & Related papers  (2024-11-11T17:10:56Z)
• CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation [49.41531871253317]
  We present a new Chinese Vision- Language Understanding Evaluation benchmark dataset. The selection of object categories and images is entirely driven by Chinese native speakers. We find that fine-tuning on Chinese culture-related VL datasets effectively enhances VLMs' understanding of Chinese culture.
  arXiv  Detail & Related papers  (2024-07-01T08:35:37Z)
• Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model [36.01840141194335]
  We introduce CT-LLM, a 2B large language model (LLM) Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by incorporating Chinese textual data. CT-LLM excels in Chinese language tasks, and showcases its adeptness in English through SFT.
  arXiv  Detail & Related papers  (2024-04-05T15:20:02Z)
• CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models [53.9835961434552]
  We introduce the Chinese Instruction-Following Benchmark (CIF-Bench) to evaluate the generalizability of large language models (LLMs) to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance.
  arXiv  Detail & Related papers  (2024-02-20T16:02:12Z)
• Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Models [89.94270049334479]
  This paper identifies a cultural dominance issue within large language models (LLMs) LLMs often provide inappropriate English-culture-related answers that are not relevant to the expected culture when users ask in non-English languages.
  arXiv  Detail & Related papers  (2023-10-19T05:38:23Z)
• CMMLU: Measuring massive multitask language understanding in Chinese [133.70911295934746]
  This paper introduces a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
  arXiv  Detail & Related papers  (2023-06-15T15:49:51Z)
• Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation [12.86275938443485]
  We examine the influence of training data factors, including quantity, quality, and linguistic distribution, on model performance. We assess various models using a evaluation set of 1,000 samples, encompassing nine real-world scenarios. We extend the vocabulary of LLaMA - the model with the closest open-source performance to proprietary language models like GPT-3.
  arXiv  Detail & Related papers  (2023-04-16T18:37:39Z)
• CPM: A Large-scale Generative Chinese Pre-trained Language Model [76.65305358932393]
  We release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning.
  arXiv  Detail & Related papers  (2020-12-01T11:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.