Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

• URL: http://arxiv.org/abs/2603.12246v1
• Date: Thu, 12 Mar 2026 17:57:06 GMT
• Title: Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
• Abstract summary: Reasoning LLMs-as-Judges provide a promising path for extending the success of reasoning models to non-verifiable domains.<n>This study investigates the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment.<n>We find that reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs.
• Score: 75.98427023288052
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning LLMs-as-Judges, which can benefit from inference-time scaling, provide a promising path for extending the success of reasoning models to non-verifiable domains where the output correctness/quality cannot be directly checked. However, while reasoning judges have shown better performance on static evaluation benchmarks, their effectiveness in actual policy training has not been systematically examined. Therefore, we conduct a rigorous study to investigate the actual impact of non-reasoning and reasoning judges in reinforcement-learning-based LLM alignment. Our controlled synthetic setting, where a "gold-standard" judge (gpt-oss-120b) provides preference annotations to train smaller judges, reveals key differences between non-reasoning and reasoning judges: non-reasoning judges lead to reward hacking easily, while reasoning judges can lead to policies that achieve strong performance when evaluated by the gold-standard judge. Interestingly, we find that the reasoning-judge-trained policies achieve such strong performance by learning to generate highly effective adversarial outputs that can also score well on popular benchmarks such as Arena-Hard by deceiving other LLM-judges. Combined with our further analysis, our study highlights both important findings and room for improvements for applying (reasoning) LLM-judges in non-verifiable LLM post-training.

Related papers

• Reinforcement Learning-based Knowledge Distillation with LLM-as-a-Judge [15.157091997511623]
  Reinforcement Learning (RL) has been shown to substantially improve the reasoning capability of small and large language models (LLMs)<n>We propose an RL framework that uses rewards from an LLM that acts as a judge evaluating model outputs over large amounts of unlabeled data.
  arXiv  Detail & Related papers  (2026-04-03T01:23:04Z)
• References Improve LLM Alignment in Non-Verifiable Domains [118.26447686644808]
  We investigate whether reference-guided LLM-evaluators can bridge the gap by serving as soft "verifiers"<n>We show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models.<n>We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges.
  arXiv  Detail & Related papers  (2026-02-18T19:03:34Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Ask a Strong LLM Judge when Your Reward Model is Uncertain [46.41334493044746]
  We propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge.<n>Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification.<n> Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost.
  arXiv  Detail & Related papers  (2025-10-23T09:09:13Z)
• Quantitative LLM Judges [60.773734899532336]
  We propose quantitative LLM judges, which align evaluation scores of existing LLM judges to humans in a given domain.<n>The models are trained to improve the score of the original judge using its rationale and score.<n>Our experiments show that quantitative judges can improve the predictive power of existing judges through post-hoc modeling.
  arXiv  Detail & Related papers  (2025-06-03T14:44:23Z)
• Reverse Engineering Human Preferences with Reinforcement Learning [15.70441204657966]
  Large Language Models (LLMs) are routinely evaluated by other LLMs trained to predict human preferences.<n>Previous work shows that the answers generated by a candidate-LLM can be edited post hoc to maximise the score assigned to them by a judge-LLM.<n>We adopt a different approach and use the signal provided by judge-LLMs as a reward to adversarially tune models.
  arXiv  Detail & Related papers  (2025-05-21T17:48:16Z)
• LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations [29.031539043555362]
  Large Language Models (LLMs) are increasingly used to evaluate information systems.<n>Recent studies suggest that LLM-based evaluations often align with human judgments.<n>This paper examines scenarios where LLM-evaluators may falsely indicate success.
  arXiv  Detail & Related papers  (2025-04-27T02:14:21Z)
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge [84.34545223897578]
  Despite their excellence in many domains, potential issues are under-explored, undermining their reliability and the scope of their utility. We identify 12 key potential biases and propose a new automated bias quantification framework-CALM- which quantifies and analyzes each type of bias in LLM-as-a-Judge. Our work highlights the need for stakeholders to address these issues and remind users to exercise caution in LLM-as-a-Judge applications.
  arXiv  Detail & Related papers  (2024-10-03T17:53:30Z)
• Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates [11.948519516797745]
  We develop an open-source framework to evaluate, compare, and visualize the reliability and alignment of LLM judges.<n>Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.
  arXiv  Detail & Related papers  (2024-08-23T11:49:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.