RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning

• URL: http://arxiv.org/abs/2605.13695v1
• Date: Wed, 13 May 2026 15:48:16 GMT
• Title: RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
• Abstract summary: RTLC promotes a single black-box LLM into an ensemble-of-thought judge with no fine-tuning, retrieval, or external tools.<n>RTLC beats N=10 self-consistency majority voting (77.7%) and a zero-shot first candidate (74.0%)<n>RTLC composes itself with post-hoc judge-score calibration, with the two interventions compounding multiplicatively in practice.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-as-a-judge is now the default measurement instrument for open-ended generation, but on the public JudgeBench benchmark even strong instruction-tuned judges barely scrape past random on objective-correctness pairwise items. We introduce RTLC, a three-stage prompting recipe -- Research, Teach-to-Learn, Critique -- that promotes a single black-box LLM into an ensemble-of-thought judge with no fine-tuning, retrieval, or external tools. Stage 1 wraps the input in a fixed pedagogical scaffold porting the Feynman Learning Technique (study $\to$ teach $\to$ find gaps $\to$ simplify) into LLM prompting. Stage 2 draws N=10 independent candidate verdicts at temperature 0.4. Stage 3 acts as its own critic, cross-comparing the candidate set against the original question to emit one critiqued verdict at temperature 0. On JudgeBench-GPT (350 hard pairwise items), Claude 3.7 Sonnet's pairwise accuracy climbs from 64.6% (single-shot vanilla prompt) to 78.6% (RTLC critique-of-10) -- an absolute 14.0-percentage-point gain. RTLC also beats N=10 self-consistency majority voting (77.7%) and a zero-shot first candidate (74.0%). A clean three-step ablation attributes +9.4 pp to the Teach-to-Learn scaffold, +3.7 pp to N=10 marginalisation, and +0.9 pp to explicit critique. We discuss the cost-accuracy frontier (RTLC sits above self-consistency at every working point), the error-budget breakdown across the four JudgeBench categories (knowledge, reasoning, math, coding), and how RTLC composes orthogonally with post-hoc judge-score calibration, with the two interventions compounding multiplicatively in practice.

Related papers

• trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories [0.0]
  Outcome-only evaluation is the production default for LLM agents.<n>We measure that blind spot where ground truth is known by construction.<n>We argue that judge evaluations must stratify recall by outcome survival.
  arXiv  Detail & Related papers  (2026-08-29T10:14:05Z)
• Learning to Follow In-Context Watermark Instructions via Self-Distillation [117.19153619258644]
  In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response.<n>We introduce $mathsfICWBench$, a benchmark of three verifiable ICW instruction families, each scored on both detectability and answer quality.<n>We propose a self-contained two-stage training method, requiring no distillation from a stronger model, no manual annotation, and no pre-existing ICW IF ability.
  arXiv  Detail & Related papers  (2026-08-29T03:42:28Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges [4.167333498582309]
  Training a language model against its own reference-free judgments assumes a model's verdict on a shown answer tracks correctness.<n>We show it fails structurally: conditioned on a candidate, a judge scores plausibility, not correctness, leaving false-positive basins a policy learns to exploit.
  arXiv  Detail & Related papers  (2026-07-07T06:59:30Z)
• Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges [2.8523456804049885]
  Safety judges are increasingly deployed to evaluate model outputs against evolving criteria.<n>Recent meta-evaluation work shows they remain brittle under prompt and rubric variation.<n>We propose a training strategy that combines instance-conditioned dynamic rubrics and a reliable-to-expressive curriculum.
  arXiv  Detail & Related papers  (2026-06-08T08:02:57Z)
• Can LLM Rerankers Predict Their Own Ranking Performance? [73.21268576961573]
  We study textitreranker-internal QPP: can an LLM reranker estimate the quality of the ranking it has just produced?<n>For training-free estimation, we examine metric-specific self-consistency across sampled rankings and verbalized confidence produced directly by the reranker.<n>Experiments on TREC Deep Learning 2019--2022 with four LLMs show that self-consistency is competitive with the state-of-the-art (SOTA) approach and better calibrated in almost all settings.<n>To improve verbalized confidence, we propose two supervised methods, Verb
  arXiv  Detail & Related papers  (2026-06-02T11:57:27Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization [0.0]
  We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure.<n>We evaluate four inference-time personalization methods across 50 authors and 1,000 generations.
  arXiv  Detail & Related papers  (2026-04-29T09:17:01Z)
• An Empirical Investigation of Practical LLM-as-a-Judge Improvement Techniques on RewardBench 2 [0.0]
  We present techniques that improve GPT-5.4 judge accuracy on RewardBench 2 without any finetuning.<n>Two techniques account for nearly all available gains: task-specific criteria injection and ensemble scoring.<n>Three further techniques did not reliably improve on criteria + ensembling at comparable cost.
  arXiv  Detail & Related papers  (2026-04-15T10:52:33Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks [12.396822247035578]
  We present exMT, a benchmark for objective extraction and metacognition.<n>Given a multi-turn transcript, a model must output a one-sentence base objective and a self-reported confidence.<n> Accuracy is scored by similarity to gold objectives, then thresholded once on 300 calibration items.
  arXiv  Detail & Related papers  (2025-08-23T03:32:04Z)
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning [54.85131761693927]
  We introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions.<n>Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards.<n>We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-05-15T14:05:15Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges [6.609843448260634]
  The LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models.<n>This paper focuses on a clean scenario in which inter-human agreement is high.<n>We identify vulnerabilities in judge models, such as their sensitivity to prompt complexity and length, and a tendency toward leniency.
  arXiv  Detail & Related papers  (2024-06-18T13:49:54Z)
• LLMs Can Patch Up Missing Relevance Judgments in Evaluation [56.51461892988846]
  We use large language models (LLMs) to automatically label unjudged documents. We simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuna-7B and GPT-3.5 Turbo respectively.
  arXiv  Detail & Related papers  (2024-05-08T00:32:19Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [48.053949045598344]
  evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively.<n>We propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks.<n>We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.