Criterion-referenceability determines LLM-as-a-judge validity across physics assessment formats

• URL: http://arxiv.org/abs/2603.14732v1
• Date: Mon, 16 Mar 2026 02:09:06 GMT
• Title: Criterion-referenceability determines LLM-as-a-judge validity across physics assessment formats
• Abstract summary: We compare GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3, and committee aggregations against human markers under blind, solution-provided, false-solution, and exemplar-anchored conditions.<n>For $n=771$ blind university exam questions, models achieve fractional mean absolute errors (fMAE) $approx 0.22$ with robust discriminative validity.<n>Across $n=55$ scripts, blind AI marking is harsher and more variable than human marking, with discriminative validity already poor.
• Score: 0.01116979912801043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models (LLMs) are increasingly considered for automated assessment and feedback, understanding when LLM marking can be trusted is essential. We evaluate LLM-as-a-judge marking across three physics assessment formats - structured questions, written essays, and scientific plots - comparing GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3, and committee aggregations against human markers under blind, solution-provided, false-solution, and exemplar-anchored conditions. For $n=771$ blind university exam questions, models achieve fractional mean absolute errors (fMAE) $\approx 0.22$ with robust discriminative validity (Spearman $ρ> 0.6$). For secondary and university structured questions ($n=1151$), providing official solutions reduces MAE and strengthens validity (committee $ρ= 0.88$); false solutions degrade absolute accuracy but leave rank ordering largely intact (committee $ρ= 0.77$; individual models $ρ\geq 0.59$). Essay marking behaves fundamentally differently. Across $n=55$ scripts ($n=275$ essays), blind AI marking is harsher and more variable than human marking, with discriminative validity already poor ($ρ\approx 0.1$). Adding a mark scheme does not improve discrimination ($ρ\approx 0$; all confidence intervals include zero). Anchored exemplars shift the AI mean close to the human mean and compress variance below the human standard deviation, but discriminative validity remains near-zero - distributional agreement can occur without valid discrimination. For code-based plot elements ($n=1400$), models achieve exceptionally high discriminative validity ($ρ> 0.84$) with near-linear calibration. Across all task types, validity tracks criterion-referenceability - the extent to which a task maps to explicit, observable grading features - and benchmark reliability, rather than raw model capability.

Related papers

• Do LLMs Have Values? A Quantitative Analysis and Alignment Framework for Values in Large Language Models [49.45276299939287]
  Large Language Models (LLMs) increasingly handle complex subjective tasks.<n>LLMs fluctuate unpredictably under minor wording changes.<n> resolving this duality is critical for reliable AI alignment.
  arXiv  Detail & Related papers  (2026-09-15T03:35:04Z)
• Evidential Rule Learning for Interpretable Classification with Abstention [3.0509197593879853]
  Fast Evidential Rule Learning (FERL) learns interpretable, accurate fuzzy rule models whose outputs are evidential.<n>FERL's belief, plausibility, and abstention capabilities arise directly from the fuzzy memberships in a single deterministic pass.
  arXiv  Detail & Related papers  (2026-08-06T10:39:48Z)
• Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation [25.235899757379844]
  Two responses can receive the same final-answer score while differing in whether the trace explicitly flags injected biasing content.<n>We introduce a minimal trace-level diagnostic with two axes: emphsusceptibility (whether the bias breaks a previously correct answer) and emphacknowledgment (whether the trace contains a rubric-defined surface reference to the injected content.
  arXiv  Detail & Related papers  (2026-06-13T05:41:57Z)
• A Spectral Phase Diagram for Binary Few-Shot Classification: Intrinsic Dimensionality, Geometric Saturation, and Representational Diagnosis [0.0]
  The saturation index $S(K) = operatornameerank(widehat_W(K)) / K$ measures the ratio of the effective rank of the pooled within-class sample covariance to the shot count.<n>The index is computable in $O(d3)$ time from support features alone, requiring no test labels or trained classifier.
  arXiv  Detail & Related papers  (2026-06-12T16:46:24Z)
• The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains [0.0]
  Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging.<n>We demonstrate that this practice produces substantially misleading rankings when two conditions co-occur.
  arXiv  Detail & Related papers  (2026-05-11T20:17:55Z)
• Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR [1.7107991816118835]
  We study the measurement risk on Japanese Financial Implicit-Commitment Recognition.<n>We find that rubric wording materially changes model-assigned labels.<n>Not every metric remains informative under the JF-ICR class distribution.
  arXiv  Detail & Related papers  (2026-04-30T03:39:14Z)
• Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection [68.37351671559675]
  Active learning can annotate thousands of instances from a short prompt at negligible cost.<n>This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be labelled at once?<n>We investigate both questions on a new dataset of 277,902 German political TikTok comments.
  arXiv  Detail & Related papers  (2026-04-15T14:10:58Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• From Global to Granular: Revealing IQA Model Performance via Correlation Surface [83.65597122328133]
  We present textbfGranularity-Modulated Correlation (GMC), which provides a structured, fine-grained analysis of IQA performance.<n>GMC includes a textbfDistribution Regulator that regularizes correlations to mitigate biases from non-uniform quality distributions.<n>Experiments on standard benchmarks show that GMC reveals performance characteristics invisible to scalar metrics, offering a more informative and reliable paradigm for analyzing, comparing, and deploying IQA models.
  arXiv  Detail & Related papers  (2026-01-29T13:55:26Z)
• Spectral Sentinel: Scalable Byzantine-Robust Decentralized Federated Learning via Sketched Random Matrix Theory on Blockchain [0.0]
  Byzantine clients poison gradients under heterogeneous (Non-IID) data.<n>We propose Spectral Sentinel, a Byzantine detection and aggregation framework.<n>We implement the full system with blockchain integration on Polygon networks.
  arXiv  Detail & Related papers  (2025-12-14T09:43:03Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models [3.938980910007962]
  Confidence calibration is an emerging challenge in real-world decision systems based on foundations models.<n>We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning.<n>We refer to it as textitconfidence misalignment penalty (CMP).
  arXiv  Detail & Related papers  (2025-01-29T11:54:37Z)
• Robust Reinforcement Learning from Corrupted Human Feedback [86.17030012828003]
  Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. We propose a robust RLHF approach -- $R3M$, which models the potentially corrupted preference label as sparse outliers. Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R3M$ improves robustness of the reward against several types of perturbations to the preference data.
  arXiv  Detail & Related papers  (2024-06-21T18:06:30Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• Evaluating the Fairness of Discriminative Foundation Models in Computer Vision [51.176061115977774]
  We propose a novel taxonomy for bias evaluation of discriminative foundation models, such as Contrastive Language-Pretraining (CLIP) We then systematically evaluate existing methods for mitigating bias in these models with respect to our taxonomy. Specifically, we evaluate OpenAI's CLIP and OpenCLIP models for key applications, such as zero-shot classification, image retrieval and image captioning.
  arXiv  Detail & Related papers  (2023-10-18T10:32:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.