Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education

• URL: http://arxiv.org/abs/2605.21614v1
• Date: Wed, 20 May 2026 18:22:22 GMT
• Title: Exploring the Effectiveness of Using LLMs for Automated Assessment of Student Self Explanations in Programming Education
• Abstract summary: Problem-solving examples could be combined with self-explanations.<n>Main challenge of this approach is assessing the correctness of the student's explanations.<n>Recent advances in automated scoring, it remains unclear whether semantic similarity methods are still the most effective technique.
• Score: 2.0515041144768134
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Worked examples are step-by-step solutions to problems in a specific domain, offered to students to acquire domain-specific problem-solving skills. The effectiveness of worked examples could be enhanced by combining them with self-explanations, which ask students to explain rather than passively study each problem-solving step. The main challenge of this approach is assessing the correctness of the student's explanations. In the prevailing approach, student explanations are judged by their semantic similarity to an instructor's or domain expert's explanation. Given recent advances in LLM-based automated scoring, it remains unclear whether semantic similarity methods are still the most effective technique to automatically score textual student responses like essays or code explanations. Comparing these methods also requires quality datasets that offer distinctive features such as balanced class distributions and domain-specific labeled data for automated scoring tasks. In this paper, we present a rigorous comparison between LLMs and semantic similarity used for automated scoring, framed as a binary classification task.

Related papers

• Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks [38.058215007885096]
  Self-evaluation for large language models (LLMs) incurs high computational overhead and introduces overconfidence issues due to intrinsic biases.<n>We propose a novel self-evaluation-free approach for unverifiable tasks, designed for lightweight yet effective self-improvement.
  arXiv  Detail & Related papers  (2025-09-27T02:44:05Z)
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [58.05959902776133]
  We introduce Single-Pass.<n>with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation.<n>We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP)<n>On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $sim$16% of training samples compared to human-labeled and other synthetically trained baselines.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)
• Improve LLM-based Automatic Essay Scoring with Linguistic Features [46.41475844992872]
  This paper develops a scoring system capable of handling essays across diverse prompts.<n>Existing methods typically fall into two categories: supervised feature-based approaches and large language model (LLM)-based methods.
  arXiv  Detail & Related papers  (2025-02-13T17:09:52Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactuals [5.565480386188035]
  Our proposed method, using counterfactual intervention assisted by Large Language Models, reveals that BERT-like models primarily focus on sentence-level features. Our approach improves understanding of neural AES methods and can also apply to other domains seeking transparency in model-driven decisions.
  arXiv  Detail & Related papers  (2024-05-29T18:16:32Z)
• Instruction Tuning with Retrieval-based Examples Ranking for Aspect-based Sentiment Analysis [7.458853474864602]
  Aspect-based sentiment analysis (ABSA) identifies sentiment information related to specific aspects and provides deeper market insights to businesses and organizations. Recent studies have proposed using fixed examples for instruction tuning to reformulate ABSA as a generation task. This study proposes an instruction learning method with retrieval-based example ranking for ABSA tasks.
  arXiv  Detail & Related papers  (2024-05-28T10:39:10Z)
• Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars [66.823588073584]
  Large language models (LLMs) have shown impressive capabilities in real-world applications. The quality of these exemplars in the prompt greatly impacts performance. Existing methods fail to adequately account for the impact of exemplar ordering on the performance.
  arXiv  Detail & Related papers  (2024-05-25T08:23:05Z)
• Automated Assessment of Students' Code Comprehension using LLMs [0.3293989832773954]
  Large Language Models (LLMs) and encoder-based Semantic Textual Similarity (STS) models are assessed. Our findings indicate that LLMs, when prompted in few-shot and chain-of-thought setting, perform comparable to fine-tuned encoder-based models in evaluating students' short answers in programming domain.
  arXiv  Detail & Related papers  (2023-12-19T20:39:12Z)
• Active Learning Principles for In-Context Learning with Large Language Models [65.09970281795769]
  This paper investigates how Active Learning algorithms can serve as effective demonstration selection methods for in-context learning. We show that in-context example selection through AL prioritizes high-quality examples that exhibit low uncertainty and bear similarity to the test examples.
  arXiv  Detail & Related papers  (2023-05-23T17:16:04Z)
• Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment [3.3064235071867856]
  Active Learning (AL) aims to reduce the labeling burden by interactively selecting the most informative samples from a pool of unlabeled data. Some studies have questioned the effectiveness of AL compared to emerging paradigms such as semi-supervised (Semi-SL) and self-supervised learning (Self-SL)
  arXiv  Detail & Related papers  (2023-01-25T15:07:44Z)
• Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
  dataset bias is one of the prevailing causes of unfairness in machine learning. We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
  arXiv  Detail & Related papers  (2021-04-14T14:20:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.