Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models

• URL: http://arxiv.org/abs/2607.28128v2
• Date: Sat, 01 Aug 2026 02:12:19 GMT
• Title: Rethinking LLM-Judged Helpfulness as a Pedagogy Signal: A Pre-Registered Audit Across Tutor Models
• Abstract summary: LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance?<n>We compare conversational and pedagogical policies instantiated with the same underlying model and paired with one weak simulated student.
• Score: 15.860351997992042
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: LLM tutoring poses a measurement problem: can a general-purpose helpfulness rubric distinguish direct answer-giving from pedagogical guidance? We audit this signal in a pre-registered study. Within each of three tutor bases, we compare conversational and pedagogical policies instantiated with the same underlying model and paired with one fixed weak simulated student. Deterministic detectors measure answer leakage and next-turn independent work. Claude Opus 4.8 is the frozen, condition-blind primary judge. After the Opus scores were fixed, GPT-5.6 Sol was prospectively specified for a post hoc robustness audit of the same 1,179 confirmatory answer-phase tutor turns under the frozen helpfulness and pedagogy rubrics. On the primary base under Opus, the policies do not differ significantly in helpfulness but are perfectly rank-separated under the pedagogy rubric (Cliff's $|δ|{=}0.10$ vs. $1.0$). Across the two judges, pedagogy contrasts retain their direction where detected, whereas the helpfulness ordering is judge-contingent, reversing between judges on two of three bases. In an Opus-only ablation, seven primary-base policies span $2.3$ points in mean judged pedagogy within a $0.25$-point band of mean judged helpfulness. Separately, answer-revealing turns are followed by less independent student work on every base, a result that is judge-invariant by construction. In this controlled setting, general-purpose helpfulness is not a reliable pedagogy signal. Tutor evaluation should pair pedagogy-targeted rubrics with deterministic process measures.

Related papers

• GradeTrap: Authority Cues in Images Shift VLM Judgments Despite Explicit Instructions to Ignore Them [0.0]
  We introduce GradeTrap, a controlled evaluation that places two social cues in direct conflict.<n>A student answer should attract sycophantic agreement, and a conflicting answer attributed to a peer, teacher, or official answer key should attract authority-based deference.<n>We test the models on 60 synthetic real-world trade-off scenarios.
  arXiv  Detail & Related papers  (2026-09-05T12:30:39Z)
• Learning to Follow In-Context Watermark Instructions via Self-Distillation [117.19153619258644]
  In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response.<n>We introduce $mathsfICWBench$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality.<n>We propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability.
  arXiv  Detail & Related papers  (2026-08-29T03:42:28Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior [0.0]
  An effective large language model (LLM) tutor must often decline to give an answer it could easily produce.<n>We report a deployed tutoring system that enforces answer-withholding as a per-turn, machine-checkable contract.<n>We tune the behavior with an automated evaluation that uses no human subjects.
  arXiv  Detail & Related papers  (2026-08-12T17:35:58Z)
• Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation [6.053609157192373]
  On-Policy Self-Distillation (OPSD) is commonly interpreted as the transfer of privileged information.<n>We investigate the role of target-specific privilege with $mathrmOP2mathrmSD$.
  arXiv  Detail & Related papers  (2026-08-10T07:51:45Z)
• OPOD: On-Policy Omni Distillation [55.440058537827134]
  Training a single model on pooled multimodal data often fails to match models specialized for individual modalities.<n>We present On-Policy Omni Distillation (OPD), which routes each student response to the matching text, image, or audio teacher.<n>OPD achieves the best average score at every scale, reaching 70.8, 51.7, and 46.2.
  arXiv  Detail & Related papers  (2026-07-23T04:55:04Z)
• H$^2$SD: Hybrid Hindsight Self-Distillation [61.71131241473028]
  Reinforcement learning with verifiable rewards (RLVR) provides reliable outcome supervision for language model reasoning.<n>Existing self-distillation methods add a privileged teacher but typically assign it a fixed role.<n>We introduce Hybrid Hindsight Self-Distillation ($mathrmH2mathrmSD$), which jointly adapts teacher context and update strategy to trajectory correctness.
  arXiv  Detail & Related papers  (2026-07-21T10:47:27Z)
• Zone of Proximal Policy Optimization: Teacher in Prompts, Not Gradients [89.89504265663057]
  Zone of Proximal Policy Optimization (ZPPO) is inspired by Vygotsky's zone of proximal development.<n>ZPPO pairs one correct teacher response with one incorrect student response as anonymized candidates the student must discriminate.<n>A prompt replay buffer recirculates each hard question until it either graduates-the student's mean rollout accuracy on it reaches half- or is FIFO-evicted.
  arXiv  Detail & Related papers  (2026-06-16T17:46:02Z)
• Measuring Whether LLM Tutors Teach or Solve: A Diagnostic for Educational Impact [4.6840536642016195]
  We study whether public tutoring benchmarks distinguish learning-supportive behavior from mere answer production.<n>We propose a lightweight diagnostic rubric based on the gap between solving-oriented and pedagogy-oriented benchmark performance.
  arXiv  Detail & Related papers  (2026-06-15T04:32:18Z)
• Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation [25.235899757379844]
  Two responses can receive the same final-answer score while differing in whether the trace explicitly flags injected biasing content.<n>We introduce a minimal trace-level diagnostic with two axes: emphsusceptibility (whether the bias breaks a previously correct answer) and emphacknowledgment (whether the trace contains a rubric-defined surface reference to the injected content.
  arXiv  Detail & Related papers  (2026-06-13T05:41:57Z)
• Surfacing Isolated Learners with Outcome-Independent Mediation of Feedback between Teachers and Students Using AI [1.6637373649145608]
  We propose a transparent mechanism that ranks course topics requiring attention without using grades or post-hoc outcome labels.<n>The approach combines three signals: student learning difficulty prevalence, disagreement between learner self-reports and observed difficulties, and unresolved teacher concerns.
  arXiv  Detail & Related papers  (2026-05-28T02:00:06Z)
• RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning [0.0]
  RTLC promotes a single black-box LLM into an ensemble-of-thought judge with no fine-tuning, retrieval, or external tools.<n>RTLC beats N=10 self-consistency majority voting (77.7%) and a zero-shot first candidate (74.0%)<n>RTLC composes itself with post-hoc judge-score calibration, with the two interventions compounding multiplicatively in practice.
  arXiv  Detail & Related papers  (2026-05-13T15:48:16Z)
• GATES: Self-Distillation under Privileged Context with Consensus Gating [89.62339954332248]
  We study self-distillation in settings where supervision is unreliable.<n>We focus on document-grounded question answering with asymmetric context.<n>We derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces.
  arXiv  Detail & Related papers  (2026-02-24T05:56:20Z)
• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision [26.922922043969958]
  We propose turning exploration into supervision through Compute as Teacher (CaT)<n>CaT synthesizes a single reference from a group of parallel rollouts and then optimizing toward it.<n>As a test-time procedure, CaT improves Gemma 3 4B, Qwen 3 4B, and Llama 3.1 8B.
  arXiv  Detail & Related papers  (2025-09-17T17:59:42Z)
• Self-Training with Differentiable Teacher [80.62757989797095]
  Self-training achieves enormous success in various semi-supervised and weakly-supervised learning tasks. The method can be interpreted as a teacher-student framework, where the teacher generates pseudo-labels, and the student makes predictions. We propose ours, short for differentiable self-training, that treats teacher-student as a Stackelberg game.
  arXiv  Detail & Related papers  (2021-09-15T02:06:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.