CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging

• URL: http://arxiv.org/abs/2605.19665v1
• Date: Tue, 19 May 2026 10:59:19 GMT
• Title: CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging
• Abstract summary: We propose a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation.<n>On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy.
• Score: 95.02210956333374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond functional correctness. While rubric-based LLM judges improve interpretability by decomposing evaluation into explicit criteria, most existing pipelines remain pointwise: they score each response independently and derive preferences by comparing aggregated scores. We show that this design is poorly matched to pairwise code preference prediction and can underperform a strong monolithic judge. We propose CriterAlign, a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation through direct criterion-level pairwise judgments, tie-driven criterion refinement, swap-consistency filtering, and final pairwise synthesis. We further introduce Human-Preference-Aligned Guidance (HPAG), synthesized offline from training examples by extracting recurring rationale gaps between human preferences and monolithic judge predictions, and injected into the criterion generator, criterion judge, and final judge. On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy, with ablations confirming the contributions of pairwise criterion design and HPAG.

Related papers

• RADAR: Rubric-Aware Dependency and Redundancy Analysis for LLM-as-Judge Evaluation [3.339065092252818]
  We introduce RADAR, a diagnostic framework for estimating behaviorally coupled evaluation criteria.<n>Given a rubric, RADAR generates targeted synthetic probes, scores each probe on all criteria, and produces a directional coupling matrix.<n>Using only a small number of probes per criterion, RADAR recovers human inter-criterion correlation structure.
  arXiv  Detail & Related papers  (2026-08-03T07:21:32Z)
• ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks [11.138973570127206]
  We propose ClaHF, a human feedback-inspired reinforcement learning framework for text classification.<n>ClaHF integrates preference modeling and RL optimization into the classification pipeline without requiring additional human annotations.<n>Results demonstrate that ClaHF consistently improves both classification performance and confidence calibration across diverse language models.
  arXiv  Detail & Related papers  (2026-05-17T14:00:01Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• Autorubric: A Unified Framework for Rubric-Based LLM Evaluation [34.429649156970015]
  We present a unified framework for evaluating large language models (LLMs)<n>Each identified technique is paired with its realization in Autorubric, an open-source Python framework proposed in this paper.<n>Autorubric supports binary, ordinal, and nominal criteria with weights; single-judge and multi-judge ensemble evaluation with majority, weighted, unanimous, and any-vote aggregation.
  arXiv  Detail & Related papers  (2026-02-13T02:26:30Z)
• Distribution-Calibrated Inference time compute for Thinking LLM-as-a-Judge [5.855996386998925]
  Thinking Large Language Models (LLMs) used as judges for pairwise preferences remain noisy at the single-sample level.<n>We study inference-time compute (ITC) for evaluators that generate n independent thinking-rating samples per item.
  arXiv  Detail & Related papers  (2025-12-02T18:46:47Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs [32.27940625341602]
  Personalised text generation is essential for user-centric information systems.<n>We introduce textbfPREF, a textbfPersonalised textbfReference-free textbfEvaluation textbfFramework.
  arXiv  Detail & Related papers  (2025-08-08T14:32:31Z)
• A Consequentialist Critique of Binary Classification Evaluation Practices [4.603739046972463]
  We show a preference for top-K metrics or fixed thresholds in evaluations at major conferences like ICML, FAccT, and CHIL.<n>We use this decision-theoretic framework to map evaluation metrics to their optimal use cases, along with a Python package, briertools, to promote the broader adoption of Brier scores.
  arXiv  Detail & Related papers  (2025-04-06T15:58:01Z)
• A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions [60.06461883533697]
  We first identify a set of fidelity criteria that reliable benchmarks for attribution methods are expected to fulfill.<n>We then introduce a Backdoor-based eXplainable AI benchmark (BackX) that adheres to the desired fidelity criteria.<n>Our analysis also offers insights into defending against neural Trojans by utilizing the attributions.
  arXiv  Detail & Related papers  (2024-05-02T13:48:37Z)
• HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition [92.17397504834825]
  HD-Eval is a framework that iteratively aligns large language models evaluators with human preference. HD-Eval inherits the essence from the evaluation mindset of human experts and enhances the alignment of LLM-based evaluators. Extensive experiments on three evaluation domains demonstrate the superiority of HD-Eval in further aligning state-of-the-art evaluators.
  arXiv  Detail & Related papers  (2024-02-24T08:01:32Z)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [69.91340332545094]
  We introduce FLASK, a fine-grained evaluation protocol for both human-based and model-based evaluation. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance.
  arXiv  Detail & Related papers  (2023-07-20T14:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.