Related papers: Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement

URL: http://arxiv.org/abs/2604.22517v1
Date: Fri, 24 Apr 2026 12:56:54 GMT
Title: Aggregate vs. Personalized Judges in Business Idea Evaluation: Evidence from Expert Disagreement
Abstract summary: Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection.<n>We then compare three judge configurations: a zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator's scoring history.<n>Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning.
Score: 4.814048071575166
License: http://creativecommons.org/licenses/by-sa/4.0/
Abstract: Evaluating LLM-generated business ideas is often harder to scale than generating them. Unlike standard NLP benchmarks, business idea evaluation relies on multi-dimensional criteria such as feasibility, novelty, differentiation, user need, and market size, and expert judgments often disagree. This paper studies a methodological question raised by such disagreement: should an automatic judge approximate an aggregate consensus, or model evaluators individually? We introduce PBIG-DATA, a dataset of approximately 3,000 individual scores across 300 patent-grounded product ideas, provided by domain experts on six business-oriented dimensions: specificity, technical validity, innovativeness, competitive advantage, need validity, and market size. Analyses show substantial expert disagreement on fine-grained ordinal scores, while agreement is higher under coarse selection, suggesting structured heterogeneity rather than random noise. We then compare three judge configurations: a rubric-only zero-shot judge, an aggregate judge conditioned on mixed evaluator histories, and a personalized judge conditioned on the target evaluator's scoring history. Across dimensions and model sizes, personalized judges align more closely with the corresponding evaluator than aggregate judges, and evaluator agreement correlates with similarity of judge-generated reasoning only under personalized conditioning. These results indicate that pooled labels can be a fragile target in pluralistic evaluation settings and motivate evaluator-conditioned judge designs for business idea assessment.

Related papers

Small Language Models as Judges for Rubric-Based Reinforcement Learning [57.707881387886516]
We study whether smaller language models can serve as efficient and reliable rubric-based judges.<n>We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges.<n>Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods.
arXiv Detail & Related papers (2026-08-30T20:00:17Z)
JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment [2.961401607740788]
We release JudgmentBench, a benchmark of 30 real-world legal tasks paired with 1,539 rubric scores and 1,530 pairwise preference judgments.<n>The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts.
arXiv Detail & Related papers (2026-05-24T19:52:39Z)
CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging [95.02210956333374]
We propose a criterion-centric framework that adapts rubric-based judging to pairwise preference evaluation.<n>On BigCodeReward, CriterAlign improves a Qwen2.5-VL-32B monolithic judge from 60.4% to 66.3% accuracy.
arXiv Detail & Related papers (2026-05-19T10:59:19Z)
Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
arXiv Detail & Related papers (2026-04-21T14:21:15Z)
Self-Preference Bias in Rubric-Based Evaluation of Large Language Models [24.994793163290737]
We present the first study of self-reference bias (SPB) in rubric evaluation.<n>We show that SPB persists even when evaluation criteria are entirely objective.<n>We analyze the factors that drive SPB in this setting, finding that negative rubrics, extreme rubric lengths, and subjective topics like emergency referrals are particularly susceptible.
arXiv Detail & Related papers (2026-04-08T12:13:53Z)
CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation [6.3121191919394475]
This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components.<n>Based on this analysis, CyclicJudge, a round-robin assignment of judges to scenarios, is demonstrated to be the optimal strategy for a fixed judge-call budget.
arXiv Detail & Related papers (2026-03-02T13:46:32Z)
EigenBench: A Comparative Behavioral Measure of Value Alignment [0.28707625120094377]
EigenBench is a black-box method for benchmarking language models' values.<n>It is designed to quantify subjective traits for which reasonable judges may disagree on the correct label.<n>It can recover model rankings on the GPQA benchmark without access to objective labels.
arXiv Detail & Related papers (2025-09-02T04:14:26Z)
CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
arXiv Detail & Related papers (2025-07-12T01:34:24Z)
J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization [69.23273504123941]
We train judges to be robust to positional biases that arise in more complex evaluation settings.<n>We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work.<n>We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%.
arXiv Detail & Related papers (2025-05-19T16:50:35Z)
Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings [36.449658676568234]
Large language model (LLM)-as-judge paradigm has been used to meet the demand for a cheap, reliable, and fast evaluation of model outputs.<n>We propose ContextualJudgeBench, a judge benchmark with 2,000 challenging response pairs across eight splits inspired by real-world contextual evaluation scenarios.<n>Our comprehensive study reveals that the contextual information and its assessment criteria present a significant challenge to even state-of-the-art models.
arXiv Detail & Related papers (2025-03-19T18:09:19Z)
Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models [68.92020689188887]
Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs)<n>Existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation.<n>This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models.
arXiv Detail & Related papers (2025-02-26T04:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.