Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams

• URL: http://arxiv.org/abs/2604.06898v1
• Date: Wed, 08 Apr 2026 09:56:31 GMT
• Title: Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams
• Abstract summary: GPT-5 performed best on English-language certifications, while Qwen-Plus performed better in Chinese contexts.<n>DeepSeek-R1 achieved the most balanced cross-lingual performance, whereas Llama-3.3 showed clear limitations in higher-order reasoning and robustness.
• Score: 34.15435577373619
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are increasingly applied in computer science education for tasks such as tutoring, content generation, and code assessment. However, systematic evaluations aligned with formal curricula and certification standards remain limited. This study benchmarked four recent models, including GPT-5, DeepSeek-R1, Qwen-Plus, and Llama-3.3-70B-Instruct, using a dataset of 1,068 questions derived from six certification exams covering networking, office applications, and Java programming. We evaluated performance across language (Chinese vs. English), cognitive levels based on Bloom's Taxonomy, domain knowledge, confidence-accuracy alignment, and robustness to input masking. Results showed that GPT-5 performed best on English-language certifications, while Qwen-Plus performed better in Chinese contexts. DeepSeek-R1 achieved the most balanced cross-lingual performance, whereas Llama-3.3 showed clear limitations in higher-order reasoning and robustness. All models performed worse on more complex tasks. These findings provide empirical support for the integration of LLMs into computer science education and offer practical implications for curriculum design and assessment.

Related papers

• Assessing the Software Security Comprehension of Large Language Models [4.1613645562134085]
  This work systematically evaluates the security comprehension of five leading large language models (LLMs)<n>We assess six cognitive dimensions: remembering, understanding, applying, analyzing, evaluating, and creating.<n>We introduce a software security knowledge boundary that identifies the highest cognitive level at which a model consistently maintains reliable performance.
  arXiv  Detail & Related papers  (2025-12-24T15:29:54Z)
• Harnessing the Power of Large Language Models for Software Testing Education: A Focus on ISTQB Syllabus [0.6524460254566903]
  The International Software Testing Qualifications Board (ISTQB) certification framework is globally recognized and widely adopted in industry and academia.<n>This paper explores and evaluates how large language models (LLMs) can complement the ISTQB framework for higher education.
  arXiv  Detail & Related papers  (2025-10-25T14:45:58Z)
• Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspective [53.594353527056775]
  We propose Chinese Commonsense Multi-hop Reasoning ( CCMOR) to evaluate Large Language Models (LLMs)<n> CCMOR is designed to evaluate LLMs' ability to integrate Chinese-specific factual knowledge with multi-step logical reasoning.<n>We implement a human-in-the-loop verification system, where domain experts systematically validate and refine the generated questions.
  arXiv  Detail & Related papers  (2025-10-09T20:29:00Z)
• CDTP: A Large-Scale Chinese Data-Text Pair Dataset for Comprehensive Evaluation of Chinese LLMs [71.01843542502438]
  We present a comprehensive benchmark for evaluating Chinese Large Language Models (CB-ECLLM)<n>CB-ECLLM is based on the newly constructed Chinese Data-Text Pair (CDTP) dataset.<n>CDTP comprises over 7 million aligned text pairs, each consisting of unstructured text coupled with one or more corresponding triples, alongside a total of 15 million triples spanning four critical domains.
  arXiv  Detail & Related papers  (2025-10-07T15:33:52Z)
• TASE: Token Awareness and Structured Evaluation for Multilingual Language Models [8.058965963418785]
  TASE is a benchmark designed to evaluate large language models' ability to perceive and reason about token-level information.<n> TASE covers 10 tasks under two core categories: token awareness and structural understanding, spanning Chinese, English, and Korean.<n>We evaluate over 30 leading commercial and open-source LLMs, including O3, Claude 4, Gemini 2.5 Pro, and DeepSeek-R1.
  arXiv  Detail & Related papers  (2025-08-07T15:11:17Z)
• Large Language Models Acing Chartered Accountancy [0.4711628883579317]
  This paper introduces CA-Ben, a Chartered Accountancy benchmark specifically designed to evaluate the financial, legal, and quantitative reasoning capabilities of LLMs.<n>Six prominent LLMs i.e. GPT 4o, LLAMA 3.3 70B, LLAMA 3.1 405B, MISTRAL Large, Claude 3.5 Sonnet, and Microsoft Phi 4 were evaluated using standardized protocols.<n>Results indicate variations in performance, with Claude 3.5 Sonnet and GPT-4o outperforming others, especially in conceptual and legal reasoning.
  arXiv  Detail & Related papers  (2025-06-26T06:10:37Z)
• XIFBench: Evaluating Large Language Models on Multilingual Instruction Following [59.549015333755186]
  Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications.<n>Existing evaluations lack fine-grained constraint analysis across diverse linguistic contexts.<n>We introduce XIFBench, a comprehensive benchmark for evaluating multilingual instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-03-10T17:07:52Z)
• LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models [46.77647640464652]
  Chinese Large Language Models (LLMs) have recently demonstrated impressive capabilities across various NLP benchmarks and real-world applications. We propose LHMKE, a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark. It encompasses 10,465 questions across 75 tasks covering 30 subjects, ranging from primary school to professional certification exams.
  arXiv  Detail & Related papers  (2024-03-19T10:11:14Z)
• Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet? [82.02076369811402]
  Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. We present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements.
  arXiv  Detail & Related papers  (2024-03-04T10:48:13Z)
• LLaMA Beyond English: An Empirical Study on Language Capability Transfer [49.298360366468934]
  We focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language. We analyze the impact of key factors such as vocabulary extension, further pretraining, and instruction tuning on transfer. We employ four widely used standardized testing benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench.
  arXiv  Detail & Related papers  (2024-01-02T06:29:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.