Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

• URL: http://arxiv.org/abs/2603.06066v1
• Date: Fri, 06 Mar 2026 09:21:51 GMT
• Title: Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring
• Abstract summary: Large Language Models (LLMs) have made it possible to evaluate student writing with unprecedented flexibility.<n>This paper investigates the application of state-of-the-art open-weight LLMs for the grading of Austrian A-level German texts.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated Essay Scoring (AES) has been explored for decades with the goal to support teachers by reducing grading workload and mitigating subjective biases. While early systems relied on handcrafted features and statistical models, recent advances in Large Language Models (LLMs) have made it possible to evaluate student writing with unprecedented flexibility. This paper investigates the application of state-of-the-art open-weight LLMs for the grading of Austrian A-level German texts, with a particular focus on rubric-based evaluation. A dataset of 101 anonymised student exams across three text types was processed and evaluated. Four LLMs, DeepSeek-R1 32b, Qwen3 30b, Mixtral 8x7b and LLama3.3 70b, were evaluated with different contexts and prompting strategies. The LLMs were able to reach a maximum of 40.6% agreement with the human rater in the rubric-provided sub-dimensions, and only 32.8% of final grades matched the ones given by a human expert. The results indicate that even though smaller models are able to use standardised rubrics for German essay grading, they are not accurate enough to be used in a real-world grading environment.

Related papers

• Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks [0.0]
  This study investigates whether large language models (LLMs) exhibit implicit grading bias based on writing style when the underlying content correctness remains constant.<n>Two state-of-the-art open-source LLMs were prompted to grade responses on a 1-10 scale with explicit instructions to evaluate content correctness only and to disregard writing style.<n>Our results reveal statistically significant grading bias in Essay/Writing tasks across both models and all perturbation types.
  arXiv  Detail & Related papers  (2026-03-19T11:20:07Z)
• Machine-Assisted Grading of Nationwide School-Leaving Essay Exams with LLMs and Statistical NLP [0.0]
  Large language models (LLMs) enable rapid and consistent automated evaluation of open-ended exam responses.<n>We operationalize the official curriculum-based rubric and compare LLM and statistical natural language processing (NLP) based assessments with human panel scores.<n>Results show that automated scoring can achieve performance comparable to that of human raters and tends to fall within the human scoring range.
  arXiv  Detail & Related papers  (2026-01-22T20:44:39Z)
• How Well Do LLMs Imitate Human Writing Style? [2.3754840025365183]
  Large language models (LLMs) can generate fluent text, but their ability to replicate the distinctive style of a specific human author remains unclear.<n>We present a fast, training-free framework for authorship verification and style imitation analysis.<n>It achieves 97.5% accuracy on academic essays and 94.5% in cross-domain evaluation.
  arXiv  Detail & Related papers  (2025-09-29T15:34:40Z)
• Can AI grade your essays? A comparative analysis of large language models and teacher ratings in multidimensional essay scoring [8.71931996488953]
  Large language models offer potential solutions to facilitate essay-scoring tasks for teachers. Recent developments in generative AI, such as large language models, offer potential solutions to facilitate essay-scoring tasks for teachers. We evaluate the performance and reliability of both open-source and closed-source LLMs in assessing German student essays.
  arXiv  Detail & Related papers  (2024-11-25T12:33:14Z)
• A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look [52.114284476700874]
  This paper reports on the results of a large-scale evaluation (the TREC 2024 RAG Track) where four different relevance assessment approaches were deployed. We find that automatically generated UMBRELA judgments can replace fully manual judgments to accurately capture run-level effectiveness. Surprisingly, we find that LLM assistance does not appear to increase correlation with fully manual assessments, suggesting that costs associated with human-in-the-loop processes do not bring obvious tangible benefits.
  arXiv  Detail & Related papers  (2024-11-13T01:12:35Z)
• Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs [3.9627148816681284]
  This article assesses which ChatGPT inputs produce better quality score estimates.<n>The optimal input is the article title and abstract, with average ChatGPT scores based on these correlating at 0.67 with human scores.
  arXiv  Detail & Related papers  (2024-08-13T09:19:21Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators.<n>The question of how reliable these evaluators are has emerged as a crucial research question.<n>We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization [66.08074487429477]
  We investigate the stability and reliability of large language models (LLMs) as automatic evaluators for abstractive summarization. We find that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements.
  arXiv  Detail & Related papers  (2023-05-22T14:58:13Z)
• Large Language Models are Diverse Role-Players for Summarization Evaluation [82.31575622685902]
  A document summary's quality can be assessed by human annotators on various criteria, both objective ones like grammar and correctness, and subjective ones like informativeness, succinctness, and appeal. Most of the automatic evaluation methods like BLUE/ROUGE may be not able to adequately capture the above dimensions. We propose a new evaluation framework based on LLMs, which provides a comprehensive evaluation framework by comparing generated text and reference text from both objective and subjective aspects.
  arXiv  Detail & Related papers  (2023-03-27T10:40:59Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases [55.45617404586874]
  We propose a few-shot instruction-based method for prompting pre-trained language models (LMs) We show that large LMs can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models.
  arXiv  Detail & Related papers  (2021-12-15T04:19:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.