Co-Evolving LLM Evaluators and Policies via DynamicRubric
Abstract Overview
This paper studies evaluator-guided post-training for large language models and argues that policy optimization depends critically on preserving relative score gaps among candidate responses sampled from the current policy. The authors propose DynamicRubric, a response-set-conditioned evaluator–policy co-evolution framework in which a generator creates weighted binary rubric items for each candidate set and a verifier aggregates the resulting judgments into response-level scores. The method is motivated by a theoretical view showing that the directional gain of shifting probability mass from one response to another equals the evaluator score gap between them, making discriminative score gaps the key optimization signal. Experiments with 8B backbones evaluate both evaluator quality and downstream policy training, and the paper also reports deployment in WeChat Search's AI answering system.
Novelty
The distinctive contribution is a dynamic, response-set-conditioned rubric evaluator that is co-evolved with the policy it supervises, rather than using a fixed prompt-only evaluator or static rubric. The paper also links evaluator design to policy optimization through a probability-allocation analysis that formalizes relative score gaps as the local training signal.
Results
With Qwen3-8B backbones, the trained DR-Generator-8B improves evaluator performance over zero-shot dynamic rubric generators, including a larger 32B variant, and is competitive with or stronger than larger scalar reward models on several preference benchmarks. As a supervision signal for policy training, DynamicRubric yields the best reported results in the paper on AlpacaEval2, ArenaHardv2.0, WildBench, and WritingBench, outperforming larger reward-model and static-rubric baselines. The optimized policy also shows gains on verifiable reasoning and coding tasks, and the paper reports statistically significant online improvements in WeChat Search before full production deployment.
Key Points
- DynamicRubric conditions evaluation on the full candidate response set, generating weighted binary rubric items that target distinctions among current-policy outputs.
- The method jointly optimizes evaluator discriminability on policy-induced samples and ranking calibration using anchor data, supporting continuous evaluator–policy co-evolution.
- Empirically, the framework improves both evaluator accuracy and downstream policy performance, with additional transfer to verifiable tasks and reported success in a large-scale production deployment.