EnGRICH: Enhancing Generative Reward Modeling with Critiques from Humans
Abstract Overview
This paper investigates how to train generative reward models (GRMs) so that their natural-language critiques are reliable rather than merely predicting correct pairwise preference judgments. The authors introduce EnGRICH, a framework that trains a GRM alongside a training-time MetaCritic initialized from a small set of human critiques and a larger set of outcome-only preference data. MetaCritic generates response-specific rubrics and verifies whether candidate GRM critiques accurately cover essential keypoints, yielding both scalar process rewards and structured diagnostic feedback. During training, these evaluative criteria are generalized to broader outcome-only data, while at inference the GRM operates independently without the MetaCritic.
Novelty
EnGRICH treats limited human critiques as a source of reusable, keypoint-level evaluative criteria rather than collapsing them into scalar rewards alone. It uniquely integrates rubric generation, keypoint verification, process reward shaping, and diagnosis-guided exploration while generalizing these criteria to outcome-only preference data without adding inference overhead.
Results
Across seven reward-model benchmarks and two controlled backbones, EnGRICH achieved the best controlled overall averages of 69.22 on Qwen3-4B and 70.32 on Gemma3-12B, outperforming GRPO, RC-GRPO, and RM-NLHF. The model also improved critique quality with higher CC-F1 and lower outcome-process inconsistency, and led to superior downstream policy optimization results across IFEval P-Strict, AlpacaEval 2 LC, and Arena-Hard v2 CW.
Key Points
- EnGRICH introduces a training-time MetaCritic that generates instance-specific rubrics and verifies critique keypoints as MATCH, MISSING, or INCORRECT to provide fine-grained process supervision.
- The framework provides a Keypoint-F1 process reward for preference-correct predictions and utilizes rubric-guided exploration to restore gradient signals when all sampled assessments in a group fail.
- Empirical evaluations show consistent gains in reward modeling accuracy, critique quality metrics, and downstream policy optimization, with the GRM operating independently at inference time.