SLMJury: Can Small Language Models Judge as Well as Large Ones?

• URL: http://arxiv.org/abs/2606.07810v1
• Date: Fri, 05 Jun 2026 19:38:15 GMT
• Title: SLMJury: Can Small Language Models Judge as Well as Large Ones?
• Abstract summary: We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms.<n>We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks.<n> Reliable automated evaluation does not require large proprietary models, yet no single SLM dominates.
• Score: 1.5990700377571574
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability. We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms: closed-ended binary correctness and open-ended quality scoring. We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks: eight closed-ended tasks spanning mathematical, scientific, and general reasoning (N=64,824 judgments per configuration), plus SummEval and MT-Bench for summarization and conversational scoring. We formalize judging as a budget-conditioned function and study five dimensions. Four findings emerge. (1) The overthinking effect is domain-dependent: for most judges quick 10-token verdicts match or beat extended reasoning on mathematical judging (by 2-7% where they help), while reasoning wins on general tasks by up to 23%. (2) Domain generalization separates model families, with math-to-general accuracy gaps ranging from under 10% to nearly 40%. (3) Closed-ended and open-ended judging draw on different capabilities: the best binary judge (Phi-4) drops to rank 9 on MT-Bench, while reasoning-trained models invert this ordering. (4) Under the Reflect-Critique-Refine (RCR) debate protocol, multi-agent debate degrades accuracy across all tested configurations, whereas the top judges resist six adversarial personas with <=0.55% variance. Reliable automated evaluation does not require large proprietary models, yet no single SLM dominates. The leaderboard is available at https://anishh15.github.io/SLMJury/, and our framework code and pip package are publicly available at https://github.com/anishh15/SLMJury and https://pypi.org/project/slmjury/.

Related papers

• Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges [31.082191748525137]
  Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.<n>We study the practical goal of identifying the topk items under a fixed comparison budget.<n>We introduce a topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty.
  arXiv  Detail & Related papers  (2026-07-02T12:39:30Z)
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels [0.0]
  LLM-as-a-judge panels aggregate votes from multiple models.<n>We quantify how far their reliability falls short of the independent-voting ideal.
  arXiv  Detail & Related papers  (2026-05-28T11:48:17Z)
• Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines [0.0]
  We compare nine debiasing strategies across five judge models from four provider families.<n>Style bias is the dominant bias (0.76-0.92 across all models), far exceeding position bias (= 0.04)<n>The combined budget strategy significantly improves Claude Sonnet 4 by +11.2 pp (p 0.0001), with directionally positive trends for other models.
  arXiv  Detail & Related papers  (2026-04-25T07:18:30Z)
• JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation [13.831735556002426]
  Small language models (SLMs) have shown promise on various reasoning tasks.<n>Their ability to judge the correctness of answers remains unclear compared to large language models (LLMs)
  arXiv  Detail & Related papers  (2025-11-20T01:14:39Z)
• Quantitative LLM Judges [60.773734899532336]
  We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain.<n>The models are trained to improve the score of the original judge using its rationale and score.<n>Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling.
  arXiv  Detail & Related papers  (2025-06-03T14:44:23Z)
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization [69.23273504123941]
  We train judges to be robust to positional biases that arise in more complex evaluation settings.<n>We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work.<n>We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%.
  arXiv  Detail & Related papers  (2025-05-19T16:50:35Z)
• JudgeLRM: Large Reasoning Models as a Judge [80.07261839142548]
  We introduce JudgeLRM, a family of judgment-oriented Large Language Models (LLMs)<n>We find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios.<n>We show that JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models.
  arXiv  Detail & Related papers  (2025-03-31T02:18:51Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges [6.609843448260634]
  The LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models.<n>This paper focuses on a clean scenario in which inter-human agreement is high.<n>We identify vulnerabilities in judge models, such as their sensitivity to prompt complexity and length, and a tendency toward leniency.
  arXiv  Detail & Related papers  (2024-06-18T13:49:54Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [48.053949045598344]
  evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively.<n>We propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks.<n>We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [76.21004582932268]
  We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Arena, a crowdsourced battle platform.
  arXiv  Detail & Related papers  (2023-06-09T05:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.