Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis

• URL: http://arxiv.org/abs/2512.14561v1
• Date: Tue, 16 Dec 2025 16:33:07 GMT
• Title: Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis
• Abstract summary: We examined agreement between large language models (LLMs) and human raters in automatic essay scoring (AES)<n>Across studies, reported LLM-human agreement was generally moderate to good, with agreement indices ranging between 0.30 and 0.80.<n>Substantial variability in agreement levels was observed across studies, reflecting differences in study-specific factors as well as the lack of standardized reporting practices.
• Score: 4.086449731896867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the growing promise of large language models (LLMs) in automatic essay scoring (AES), empirical findings regarding their reliability compared to human raters remain mixed. Following the PRISMA 2020 guidelines, we synthesized 65 published and unpublished studies from January 2022 to August 2025 that examined agreement between LLMs and human raters in AES. Across studies, reported LLM-human agreement was generally moderate to good, with agreement indices (e.g., Quadratic Weighted Kappa, Pearson correlation, and Spearman's rho) mostly ranging between 0.30 and 0.80. Substantial variability in agreement levels was observed across studies, reflecting differences in study-specific factors as well as the lack of standardized reporting practices. Implications and directions for future research are discussed.

Related papers

• GLARE: Generative Learning via Adversarial Reward Estimation For Social Dynamics Forecasting [64.66483488975663]
  We introduce the Meeting Dynamic Forecasting Benchmark (MDFB)<n>It was constructed from 2,207 real-world meetings and 24,794 future-facing queries.<n>We also present GLARE, an adaptation of adversarial imitation learning to conditional language generation.
  arXiv  Detail & Related papers  (2026-09-10T19:52:34Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• Rethinking Ground Truth: A Case Study on Human Label Variation in MLLM Benchmarking [0.03219669032304847]
  Human Label Variation (HLV) remains underexplored in benchmarks despite rapid progress in large language model development.<n>We introduce an evaluation protocol for multimodal large language model benchmarking that explicitly accounts for two conditions: (1) human label agreement and (2) disagreement.
  arXiv  Detail & Related papers  (2026-03-20T08:29:05Z)
• Agreement Between Large Language Models, Human Reviewers, and Authors in Evaluating STROBE Checklists for Observational Studies in Rheumatology [0.0]
  This study compares STROBE assessments from large language models (LLMs), a human reviewer panel, and the original manuscript authors in observational rheumatology research.
  arXiv  Detail & Related papers  (2026-03-12T19:56:46Z)
• Evaluating Large Language Models for Abstract Evaluation Tasks: An Empirical Study [1.412242138378466]
  Large language models (LLMs) can process requests and generate texts, but their feasibility for assessing academic content needs further investigation.<n>This study examined ChatGPT-5, Gemini-3-Pro, and Claude-Sonnet-4.5's consistency and reliability in evaluating abstracts compared to one another and to human reviewers.
  arXiv  Detail & Related papers  (2026-01-09T15:21:17Z)
• Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency [12.62715115816099]
  Review of 63 studies published between 2023 and 2025 in leading NLP and AI venues.<n>We show that task and population of interest are often underspecified in persona-based experiments.<n>We introduce a persona transparency checklist that emphasizes representative sampling, explicit grounding in empirical data, and enhanced ecological validity.
  arXiv  Detail & Related papers  (2025-11-29T12:27:34Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• The Hidden Link Between RLHF and Contrastive Learning [56.45346439723488]
  We show that Reinforcement Learning from Human Feedback (RLHF) and the simple Direct Preference Optimization (DPO) can be interpreted from the perspective of mutual information (MI)<n>Within this framework, both RLHF and DPO can be interpreted as methods that performing contrastive learning based on the positive and negative samples derived from base model.<n>We propose the Mutual Information Optimization (MIO) to mitigate the late-stage decline in chosen-likelihood observed in DPO.
  arXiv  Detail & Related papers  (2025-06-27T18:51:25Z)
• ChatGPT for automated grading of short answer questions in mechanical ventilation [0.0]
  Large language models (LLMs) simulate conversational language and interpret unstructured free-text responses.<n>We evaluated ChatGPT 4o to grade SAQs in a postgraduate medical setting using data from 215 students.
  arXiv  Detail & Related papers  (2025-05-05T19:04:25Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments [0.7852714805965528]
  We develop a set of 30 counterfactual scenarios and collect ratings across 8 evaluation metrics from 206 respondents.<n>We fine-tuned different Large Language Models to predict average or individual human judgment across these metrics.
  arXiv  Detail & Related papers  (2024-10-28T15:33:37Z)
• A Survey on Human Preference Learning for Large Language Models [81.41868485811625]
  The recent surge of versatile large language models (LLMs) largely depends on aligning increasingly capable foundation models with human intentions by preference learning. This survey covers the sources and formats of preference feedback, the modeling and usage of preference signals, as well as the evaluation of the aligned LLMs.
  arXiv  Detail & Related papers  (2024-06-17T03:52:51Z)
• Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity [3.435381469869212]
  Sentence representation methods have reached an average Spearman's correlation score of approximately 86 across seven STS benchmarks in SentEval. We propose Pcc-tuning, which employs Pearson's correlation coefficient as a loss function to refine model performance beyond contrastive learning. Experimental results demonstrate that Pcc-tuning can markedly surpass previous state-of-the-art strategies with only a minimal amount of fine-grained annotated samples.
  arXiv  Detail & Related papers  (2024-06-14T07:40:07Z)
• DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators.<n>The question of how reliable these evaluators are has emerged as a crucial research question.<n>We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models [91.05820785008527]
  We propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) We build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment.
  arXiv  Detail & Related papers  (2023-07-20T17:59:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.