Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors

• URL: http://arxiv.org/abs/2603.25674v1
• Date: Thu, 26 Mar 2026 17:29:20 GMT
• Title: Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors
• Abstract summary: This study investigates the effects of construct-irrelevant factors on a dual-architecture LLM-based scoring system.<n>It was found that the scoring system was generally robust to padding responses with meaningless text, spelling errors, and writing sophistication.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated systems have been widely adopted across the educational testing industry for open-response assessment and essay scoring. These systems commonly achieve performance levels comparable to or superior than trained human raters, but have frequently been demonstrated to be vulnerable to the influence of construct-irrelevant factors (i.e., features of responses that are unrelated to the construct assessed) and adversarial conditions. Given the rising usage of large language models in automated scoring systems, there is a renewed focus on ``hallucinations'' and the robustness of these LLM-based automated scoring approaches to construct-irrelevant factors. This study investigates the effects of construct-irrelevant factors on a dual-architecture LLM-based scoring system designed to score short essay-like open-response items in a situational judgment test. It was found that the scoring system was generally robust to padding responses with meaningless text, spelling errors, and writing sophistication. Duplicating large passages of text resulted in lower scores predicted by the system, on average, contradicting results from previous studies of non-LLM-based scoring systems, while off-topic responses were heavily penalized by the scoring system. These results provide encouraging support for the robustness of future LLM-based scoring systems when designed with construct relevance in mind.

Related papers

• From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment [12.134587575174633]
  We propose a general framework for sentence-level interpretability of rubric-based scoring.<n>We show that SHAP provides more faithful and transferable explanations for rubric-based scoring.
  arXiv  Detail & Related papers  (2026-04-18T14:27:51Z)
• Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams [1.6955250288861923]
  Student responses in STEM assessments are often handwritten and combine symbolic expressions, calculations, and diagrams.<n>Recent advances in large language models (LLMs) have increased attention to AI-assisted scoring.<n>This study examined the reliability of AI-assisted scoring of undergraduate physics constructed responses using GPT-4o.
  arXiv  Detail & Related papers  (2026-04-14T03:04:44Z)
• Pitfalls in Evaluating Interpretability Agents [91.49742416116635]
  We build an agentic system in which a research agent iteratively designs experiments and refines hypotheses.<n>Our work demonstrates fundamental challenges in evaluating complex automated interpretability systems.
  arXiv  Detail & Related papers  (2026-03-20T16:27:17Z)
• Comparison of Scoring Rationales Between Large Language Models and Human Raters [3.4283859937936705]
  This study investigates the rationales of human and LLM raters to identify potential causes of scoring inconsistency.<n>Using essays from a large-scale test, the scoring accuracy of GPT-4o, Gemini, and other LLMs is examined.<n>Cosine similarity is used to evaluate the similarity of the rationales provided.
  arXiv  Detail & Related papers  (2025-09-27T16:58:51Z)
• RAFFLES: Reasoning-based Attribution of Faults for LLM Systems [4.950398945013938]
  RAFFLES is an evaluation architecture that incorporates reasoning and iterative refinement.<n>RAFFLES operates as an iterative, multi-component pipeline, using a central Judge to systematically investigate faults.<n>Results demonstrate a key step towards introducing automated fault detection for autonomous systems.
  arXiv  Detail & Related papers  (2025-09-08T15:57:14Z)
• RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning [64.46921169261852]
  RAG-Zeval is a novel end-to-end framework that formulates faithfulness and correctness evaluation as a rule-guided reasoning task.<n>Our approach trains evaluators with reinforcement learning, facilitating compact models to generate comprehensive and sound assessments.<n>Experiments demonstrate RAG-Zeval's superior performance, achieving the strongest correlation with human judgments.
  arXiv  Detail & Related papers  (2025-05-28T14:55:33Z)
• Limitations of Automatic Relevance Assessments with Large Language Models for Fair and Reliable Retrieval Evaluation [2.9180406633632523]
  Large language models (LLMs) are gaining much attention as tools for automatic relevance assessment.<n>Recent research has shown that LLM-based assessments yield high systems ranking correlation with human-made judgements.<n>We look at how LLM-generated judgements preserve ranking differences among top-performing systems and also how they preserve pairwise significance evaluation as human judgements.
  arXiv  Detail & Related papers  (2024-11-20T11:19:35Z)
• Auto-PRE: An Automatic and Cost-Efficient Peer-Review Framework for Language Generation Evaluation [52.76508734756661]
  Auto-PRE is an automatic evaluation framework inspired by the peer review process.<n>Unlike previous approaches that rely on human annotations, Auto-PRE automatically selects evaluators based on three core traits.<n> Experiments on three representative tasks, including summarization, non-factoid QA, and dialogue generation, demonstrate that Auto-PRE achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-10-16T06:06:06Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• HGOT: Hierarchical Graph of Thoughts for Retrieval-Augmented In-Context Learning in Factuality Evaluation [20.178644251662316]
  We introduce the hierarchical graph of thoughts (HGOT) to enhance the retrieval of pertinent passages during in-context learning. The framework employs the divide-and-conquer strategy to break down complex queries into manageable sub-queries. It refines self-consistency majority voting for answer selection, which incorporates the recently proposed citation recall and precision metrics.
  arXiv  Detail & Related papers  (2024-02-14T18:41:19Z)
• Perspectives on Large Language Models for Relevance Judgment [56.935731584323996]
  Large language models (LLMs) claim that they can assist with relevance judgments. It is not clear whether automated judgments can reliably be used in evaluations of retrieval systems.
  arXiv  Detail & Related papers  (2023-04-13T13:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.