Edu-Values: Towards Evaluating the Chinese Education Values of Large Language Models

• URL: http://arxiv.org/abs/2409.12739v2
• Date: Thu, 10 Oct 2024 06:58:35 GMT
• Title: Edu-Values: Towards Evaluating the Chinese Education Values of Large Language Models
• Authors: Peiyi Zhang, Yazhou Zhang, Bo Wang, Lu Rong, Jing Qin,
• Abstract summary: We present Edu-Values, the first Chinese education values evaluation benchmark designed to measure large language models' alignment ability. We meticulously design and compile 1,418 questions, including multiple-choice, multi-modal question answering, subjective analysis, adversarial prompts, and questions on traditional Chinese culture. Due to differences in educational culture, Chinese LLMs significantly outperform English LLMs, with Qwen 2 ranking the first with a score of 81.37.
• Score: 9.761584874383873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the recent evolution of large language models (LLMs), concerns about aligning such models with human values have grown. Previous research has primarily focused on assessing LLMs' performance in terms of the Helpful, Honest, Harmless (3H) basic principles, while often overlooking their alignment with educational values in the Chinese context. To fill this gap, we present Edu-Values, the first Chinese education values evaluation benchmark designed to measure LLMs' alignment ability across seven dimensions: professional ideology, cultural literacy, educational knowledge and skills, education laws and regulations, teachers' professional ethics, basic competencies, and subject knowledge. We meticulously design and compile 1,418 questions, including multiple-choice, multi-modal question answering, subjective analysis, adversarial prompts, and questions on traditional Chinese culture. We conduct both human evaluation and automatic evaluation over 11 state-of-the-art (SoTA) LLMs, and highlight three main findings: (1) due to differences in educational culture, Chinese LLMs significantly outperform English LLMs, with Qwen 2 ranking the first with a score of 81.37; (2) LLMs perform well in subject knowledge and teaching skills but struggle with teachers' professional ethics and basic competencies; (3) LLMs excel at multiple-choice questions but perform poorly on subjective analysis and multi-modal tasks. This demonstrates the effectiveness and potential of the proposed benchmark. Our dataset is available at https://github.com/zhangpeii/Edu-Values.git.

Related papers

• Dr.Academy: A Benchmark for Evaluating Questioning Capability in Education for Large Language Models [30.759154473275043]
  This study introduces a benchmark to evaluate the questioning capability in education as a teacher of large language models (LLMs) We apply four metrics, including relevance, coverage, representativeness, and consistency, to evaluate the educational quality of LLMs' outputs. Our results indicate that GPT-4 demonstrates significant potential in teaching general, humanities, and science courses; Claude2 appears more apt as an interdisciplinary teacher.
  arXiv  Detail & Related papers  (2024-08-20T15:36:30Z)
• FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models [64.11333762954283]
  This paper introduces FoundaBench, a pioneering benchmark designed to rigorously evaluate the fundamental knowledge capabilities of Chinese LLMs. We present an extensive evaluation of 12 state-of-the-art LLMs using FoundaBench, employing both traditional assessment methods and our CircularEval protocol to mitigate potential biases in model responses. Our results highlight the superior performance of models pre-trained on Chinese corpora, and reveal a significant disparity between models' reasoning and memory recall capabilities.
  arXiv  Detail & Related papers  (2024-04-29T01:49:07Z)
• Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches [69.73783026870998]
  This work proposes a novel framework, ValueLex, to reconstruct Large Language Models' unique value system from scratch. Based on Lexical Hypothesis, ValueLex introduces a generative approach to elicit diverse values from 30+ LLMs. We identify three core value dimensions, Competence, Character, and Integrity, each with specific subdimensions, revealing that LLMs possess a structured, albeit non-human, value system.
  arXiv  Detail & Related papers  (2024-04-19T09:44:51Z)
• Measuring Taiwanese Mandarin Language Understanding [24.581360653015423]
  We present TMLU, a holistic evaluation suit tailored for assessing the advanced knowledge and reasoning capability in large language models (LLMs) TMLU consists of an array of 37 subjects across social science, STEM, humanities, Taiwan-specific content, and others, ranging from middle school to professional levels.
  arXiv  Detail & Related papers  (2024-03-29T13:56:21Z)
• LHMKE: A Large-scale Holistic Multi-subject Knowledge Evaluation Benchmark for Chinese Large Language Models [46.77647640464652]
  Chinese Large Language Models (LLMs) have recently demonstrated impressive capabilities across various NLP benchmarks and real-world applications. We propose LHMKE, a Large-scale, Holistic, and Multi-subject Knowledge Evaluation benchmark. It encompasses 10,465 questions across 75 tasks covering 30 subjects, ranging from primary school to professional certification exams.
  arXiv  Detail & Related papers  (2024-03-19T10:11:14Z)
• E-EVAL: A Comprehensive Chinese K-12 Education Evaluation Benchmark for Large Language Models [44.74622336775077]
  We introduce the E-EVAL, the first comprehensive evaluation benchmark specifically designed for the Chinese K-12 education field. The E-EVAL consists of 4,351 multiple-choice questions at the primary, middle, and high school levels across a wide range of subjects, including Chinese, English, Politics, History, Ethics, Physics, Chemistry, Mathematics, and Geography. Findings show that Chinese-dominant models perform well compared to English-dominant models, with many scoring even above the GPT 4.0. However, almost all models perform poorly in complex subjects such as mathematics.
  arXiv  Detail & Related papers  (2024-01-29T07:34:37Z)
• ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models [17.562961249150295]
  We propose the ZhuJiu benchmark for large language models (LLMs) evaluation. ZhuJiu is the pioneering benchmark that fully assesses LLMs in Chinese, while also providing equally robust evaluation abilities in English. The ZhuJiu benchmark and open-participation leaderboard are publicly released at http://www.zhujiu-benchmark.com/.
  arXiv  Detail & Related papers  (2023-08-28T06:56:44Z)
• CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility [62.74405775089802]
  We present CValues, the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs. As a result, we have manually collected adversarial safety prompts across 10 scenarios and induced responsibility prompts from 8 domains. Our findings suggest that while most Chinese LLMs perform well in terms of safety, there is considerable room for improvement in terms of responsibility.
  arXiv  Detail & Related papers  (2023-07-19T01:22:40Z)
• A Survey on Evaluation of Large Language Models [87.60417393701331]
  Large language models (LLMs) are gaining increasing popularity in both academia and industry. This paper focuses on three key dimensions: what to evaluate, where to evaluate, and how to evaluate.
  arXiv  Detail & Related papers  (2023-07-06T16:28:35Z)
• CMMLU: Measuring massive multitask language understanding in Chinese [133.70911295934746]
  This paper introduces a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
  arXiv  Detail & Related papers  (2023-06-15T15:49:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.