Small Language Models as Judges for Rubric-Based Reinforcement Learning
Abstract Overview
This paper investigates whether small language models can serve as efficient and reliable judges for rubric-based reinforcement learning, where responses are evaluated against instance-specific criteria rather than exact-match verifiers. To evaluate this systematically, the authors construct two pointwise rubric-judging benchmarks, PointRubric and RaR-Science-Static, which provide explicit criteria, candidate responses, criterion-level labels, and weighted aggregation rules. They compare three readout methods for extracting criterion judgments from small Qwen3 backbones: generative verdicts, Yes/No log-probability scoring, and hidden-state probe classifiers. The findings demonstrate that probe readouts from frozen backbones recover evaluative signals more reliably than generative or logprob methods, especially in the science domain. Finally, applying a Qwen3-1.7B probe judge as an online reward model in reinforcement learning demonstrates improved policy performance and significant computational efficiency over a larger generative judge baseline.
Novelty
The paper introduces the formulation and empirical evaluation of small language models as criterion-level rubric judges for reinforcement learning using frozen-backbone linear probe readouts instead of standard generation-based judging. It also provides two new benchmark resources specifically structured for pointwise rubric evaluation and directly links static rubric agreement to downstream reinforcement learning policy optimization.
Results
On RaR-Science-Static, the Qwen3-1.7B probe achieves 0.835 macro-F1, substantially exceeding generative (0.443) and logprob (0.449) readouts on the same backbone. Used as a GRPO reward model, the 1.7B probe improves actor rubric scores from 0.232 to 0.643, outperforming an 8B generative judge baseline (0.594) while requiring 10.7 times less cumulative judge time. The probe-trained policy transfers to GPQA-Diamond, improving accuracy from 0.335 to 0.388, and a science-trained probe transfers zero-shot to RaR-Medicine with 0.718 macro-F1.
Key Points
- The authors construct PointRubric and RaR-Science-Static to provide pointwise rubric judging benchmarks with explicit criteria, per-criterion labels, and weighted score aggregation.
- Probe-based readouts on frozen representations from small language models reliably outperform generative and log-probability scoring for criterion-level rubric evaluation.
- A Qwen3-1.7B probe judge functions as an effective reinforcement learning reward model, producing higher policy scores than an 8B generative judge while using 10.7 times less cumulative judge compute.