Large Language Models as Partners in Student Essay Evaluation

• URL: http://arxiv.org/abs/2405.18632v1
• Date: Tue, 28 May 2024 22:28:50 GMT
• Title: Large Language Models as Partners in Student Essay Evaluation
• Authors: Toru Ishida, Tongxi Liu, Hailong Wang, William K. Cheung,
• Abstract summary: This paper presents an evaluation conducted with Large Language Models (LLMs) using actual student essays in three scenarios. Quantitative analysis of the results revealed a strong correlation between LLM and faculty member assessments in the pairwise comparison scenario with pre-specified rubrics.
• Score: 5.479797073162603
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As the importance of comprehensive evaluation in workshop courses increases, there is a growing demand for efficient and fair assessment methods that reduce the workload for faculty members. This paper presents an evaluation conducted with Large Language Models (LLMs) using actual student essays in three scenarios: 1) without providing guidance such as rubrics, 2) with pre-specified rubrics, and 3) through pairwise comparison of essays. Quantitative analysis of the results revealed a strong correlation between LLM and faculty member assessments in the pairwise comparison scenario with pre-specified rubrics, although concerns about the quality and stability of evaluations remained. Therefore, we conducted a qualitative analysis of LLM assessment comments, showing that: 1) LLMs can match the assessment capabilities of faculty members, 2) variations in LLM assessments should be interpreted as diversity rather than confusion, and 3) assessments by humans and LLMs can differ and complement each other. In conclusion, this paper suggests that LLMs should not be seen merely as assistants to faculty members but as partners in evaluation committees and outlines directions for further research.

Related papers

• From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Management [6.70908766695241]
  This study explores the potential of Large Language Models (LLMs), specifically GPT-4, to enhance objectivity in organizational task performance evaluations. Our results suggest that GPT ratings are comparable to human ratings but exhibit higher consistency and reliability. Our research suggests that while LLMs are capable of extracting meaningful constructs from text-based data, their scope is currently limited to specific forms of performance evaluation.
  arXiv  Detail & Related papers  (2024-08-09T20:35:10Z)
• Analyzing Large Language Models for Classroom Discussion Assessment [0.8194411391152481]
  We examine how the assessment performance of 2 large language models (LLMs) interacts with 3 factors that may affect performance: task formulation, context length, and few-shot examples. Our results suggest that the 3 aforementioned factors do affect the performance of the tested LLMs and there is a relation between consistency and performance. We recommend a LLM-based assessment approach that has a good balance in terms of predictive performance, computational efficiency, and consistency.
  arXiv  Detail & Related papers  (2024-06-12T22:43:38Z)
• Facilitating Holistic Evaluations with LLMs: Insights from Scenario-Based Experiments [0.22499166814992438]
  Even experienced faculty teams find it challenging to realize a holistic evaluation that accommodates diverse perspectives. This paper explores the use of a Large Language Model (LLM) as a facilitator to integrate diverse faculty assessments.
  arXiv  Detail & Related papers  (2024-05-28T01:07:06Z)
• Decompose and Aggregate: A Step-by-Step Interpretable Evaluation Framework [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators. The question of how reliable these evaluators are has emerged as a crucial research question. We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• Large Language Models are Inconsistent and Biased Evaluators [2.136983452580014]
  We show that Large Language Models (LLMs) are biased evaluators as they exhibit familiarity bias and show skewed distributions of ratings. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality.
  arXiv  Detail & Related papers  (2024-05-02T20:42:28Z)
• Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators [48.54465599914978]
  Large Language Models (LLMs) have demonstrated promising capabilities in assessing the quality of generated natural language. LLMs still exhibit biases in evaluation and often struggle to generate coherent evaluations that align with human assessments. We introduce Pairwise-preference Search (PairS), an uncertainty-guided search method that employs LLMs to conduct pairwise comparisons and efficiently ranks candidate texts.
  arXiv  Detail & Related papers  (2024-03-25T17:11:28Z)
• Evaluating Large Language Models at Evaluating Instruction Following [54.49567482594617]
  We introduce a challenging meta-evaluation benchmark, LLMBar, designed to test the ability of an LLM evaluator in discerning instruction-following outputs. We discover that different evaluators exhibit distinct performance on LLMBar and even the highest-scoring ones have substantial room for improvement.
  arXiv  Detail & Related papers  (2023-10-11T16:38:11Z)
• A Survey on Evaluation of Large Language Models [87.60417393701331]
  Large language models (LLMs) are gaining increasing popularity in both academia and industry. This paper focuses on three key dimensions: what to evaluate, where to evaluate, and how to evaluate.
  arXiv  Detail & Related papers  (2023-07-06T16:28:35Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.