Can GPT-4 do L2 analytic assessment?

• URL: http://arxiv.org/abs/2404.18557v1
• Date: Mon, 29 Apr 2024 10:00:00 GMT
• Title: Can GPT-4 do L2 analytic assessment?
• Abstract summary: Automated essay scoring (AES) to evaluate second language (L2) proficiency has been a firmly established technology used in educational contexts for decades. In this paper, we perform a series of experiments using GPT-4 in a zero-shot fashion on a publicly available dataset annotated with holistic scores. We observe significant correlations between the automatically predicted analytic scores and multiple features associated with the individual proficiency components.
• Score: 34.445391091278786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated essay scoring (AES) to evaluate second language (L2) proficiency has been a firmly established technology used in educational contexts for decades. Although holistic scoring has seen advancements in AES that match or even exceed human performance, analytic scoring still encounters issues as it inherits flaws and shortcomings from the human scoring process. The recent introduction of large language models presents new opportunities for automating the evaluation of specific aspects of L2 writing proficiency. In this paper, we perform a series of experiments using GPT-4 in a zero-shot fashion on a publicly available dataset annotated with holistic scores based on the Common European Framework of Reference and aim to extract detailed information about their underlying analytic components. We observe significant correlations between the automatically predicted analytic scores and multiple features associated with the individual proficiency components.

Related papers

• ICLE++: Modeling Fine-Grained Traits for Holistic Essay Scoring [16.481654889824448]
  We introduce ICLE++, a corpus of persuasive student essays annotated with both holistic scores and trait-specific scores.<n>Not only can ICLE++ be used to test the generalizability of AES models trained on ASAP, but it can also facilitate the evaluation of models developed for newer AES problems.
  arXiv  Detail & Related papers  (2026-07-30T04:35:02Z)
• Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs [1.3819918262591997]
  rank-based correlation metrics obscure intrinsic intercorrelations among analytic dimensions.<n>High correlations may mask a system's true diagnostic behaviour.<n>We propose a self-referential assessment framework that focuses on identifying intra-learner strengths and weaknesses.
  arXiv  Detail & Related papers  (2026-05-05T21:05:25Z)
• Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams [1.6955250288861923]
  Student responses in STEM assessments are often handwritten and combine symbolic expressions, calculations, and diagrams.<n>Recent advances in large language models (LLMs) have increased attention to AI-assisted scoring.<n>This study examined the reliability of AI-assisted scoring of undergraduate physics constructed responses using GPT-4o.
  arXiv  Detail & Related papers  (2026-04-14T03:04:44Z)
• Beyond Accuracy: Towards a Robust Evaluation Methodology for AI Systems for Language Education [0.0]
  The rapid adoption of large language models in AI-powered language education has created an urgent need for evaluations that assess pedagogical effectiveness.<n>We introduce L2-Bench, a novel evaluation benchmark grounded in a validated "language learning experience designer" construct.<n>Our methodology integrates pedagogical theory, sociotechnical AI evaluation methods, and operationalizes a hierarchical taxonomy to structure an expert-curated dataset.
  arXiv  Detail & Related papers  (2026-03-20T16:13:03Z)
• Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLP [0.0]
  Large language models (LLMs) enable rapid and consistent automated evaluation of open-ended exam responses.<n>We operationalize the official curriculum-based rubric and compare LLM and statistical natural language processing (NLP) based assessments with human panel scores.<n>Results show that automated scoring can achieve performance comparable to that of human raters and tends to fall within the human scoring range.
  arXiv  Detail & Related papers  (2026-01-22T20:44:39Z)
• Exploring LLM Autoscoring Reliability in Large-Scale Writing Assessments Using Generalizability Theory [2.5163150839708948]
  This study investigates the estimation of reliability for large language models (LLMs) in scoring writing tasks from the AP Chinese Language and Culture exam.<n>Using generalizability theory, the research evaluates and compares score consistency between human and AI raters.<n> Composite scoring that incorporates both human and AI raters improved reliability, which supports that hybrid scoring models may offer benefits for large-scale writing assessments.
  arXiv  Detail & Related papers  (2025-07-26T15:33:05Z)
• Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework [61.38174427966444]
  Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios. Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models. We propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses.
  arXiv  Detail & Related papers  (2025-02-26T06:31:45Z)
• Evaluating AI-Generated Essays with GRE Analytical Writing Assessment [15.993966092824335]
  This study examines essays generated by ten leading LLMs for the analytical writing assessment of the Graduate Record Exam (GRE) We assessed these essays using both human raters and the e-rater automated scoring engine as used in the GRE scoring pipeline. The top-performing Gemini and GPT-4o received an average score of 4.78 and 4.67, respectively.
  arXiv  Detail & Related papers  (2024-10-22T21:30:58Z)
• Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation [50.60733773088296]
  We conduct a comprehensive human evaluation of the results of several shared tasks from the last International Workshop on Spoken Language Translation (IWSLT 2023) We propose an effective evaluation strategy based on automatic resegmentation and direct assessment with segment context. Our analysis revealed that: 1) the proposed evaluation strategy is robust and scores well-correlated with other types of human judgements; 2) automatic metrics are usually, but not always, well-correlated with direct assessment scores; and 3) COMET as a slightly stronger automatic metric than chrF.
  arXiv  Detail & Related papers  (2024-06-06T09:18:42Z)
• Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models [21.280725490520798]
  This paper focuses on the ability of large language models to verify public health claims. We examine the effectiveness of zero/few-shot prompting and parameter-efficient fine-tuning across various open and closed-source models.
  arXiv  Detail & Related papers  (2024-05-15T15:49:06Z)
• Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models [92.66784679667441]
  Prometheus 2 is a more powerful evaluator LM that closely mirrors human and GPT-4 judgements. It is capable of processing both direct assessment and pairwise ranking formats grouped with a user-defined evaluation criteria. On four direct assessment benchmarks and four pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and agreement with humans and proprietary LM judges.
  arXiv  Detail & Related papers  (2024-05-02T17:59:35Z)
• From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews [14.135107583299277]
  This study explores the integration of Large Language Models (LLMs) with human expertise to enhance text analysis of stakeholder interviews regarding K-12 education policy within one U.S. state. Using a mixed-methods approach, human experts developed a codebook and coding processes as informed by domain knowledge and unsupervised topic modeling results. Results reveal that while GPT-4 thematic coding aligned with human coding by 77.89% at specific themes, expanding to broader themes increased congruence to 96.02%, surpassing traditional Natural Language Processing (NLP) methods by over 25%.
  arXiv  Detail & Related papers  (2023-12-02T18:55:14Z)
• SOUL: Towards Sentiment and Opinion Understanding of Language [96.74878032417054]
  We propose a new task called Sentiment and Opinion Understanding of Language (SOUL) SOUL aims to evaluate sentiment understanding through two subtasks: Review (RC) and Justification Generation (JG)
  arXiv  Detail & Related papers  (2023-10-27T06:48:48Z)
• A Large Language Model Approach to Educational Survey Feedback Analysis [0.0]
  This paper assesses the potential for the large language models (LLMs) GPT-4 and GPT-3.5 to aid in deriving insight from education feedback surveys.
  arXiv  Detail & Related papers  (2023-09-29T17:57:23Z)
• Large Language Models on Wikipedia-Style Survey Generation: an Evaluation in NLP Concepts [21.150221839202878]
  Large Language Models (LLMs) have achieved significant success across various general tasks. In this work, we examine the proficiency of LLMs in generating succinct survey articles specific to the niche field of NLP in computer science. We compare both human and GPT-based evaluation scores and provide in-depth analysis.
  arXiv  Detail & Related papers  (2023-08-21T01:32:45Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems [64.4896118325552]
  We evaluate the current state-of-the-art AES models using a model adversarial evaluation scheme and associated metrics. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models.
  arXiv  Detail & Related papers  (2020-07-14T03:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.