Dynamically Allocating Evaluation Effort for Model Ranking

• URL: http://arxiv.org/abs/2608.03437v1
• Date: Tue, 04 Aug 2026 10:33:04 GMT
• Title: Dynamically Allocating Evaluation Effort for Model Ranking
• Abstract summary: We formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms.<n>We show that it improves discrimination between top-performing models.<n>This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
• Score: 60.00410111594536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.

Related papers

• To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAG [31.664195597760422]
  Multi-agent document assessment for retrieval-augmented generation is computationally expensive.<n>We conduct a controlled study of training-free interventions on 7B-9B instruction-tuned models across diverse QA benchmarks.<n>We propose MADARA, a model-adaptive routing architecture.
  arXiv  Detail & Related papers  (2026-06-23T21:34:23Z)
• Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm that Provably Exploits Model Similarity [14.830460628644715]
  Large Language Models are typically benchmarked by evaluating every model on every test query.<n>We propose Synchronized Successive Rejects (SySRs), augmenting the classical Successive Rejects algorithm with paired comparisons.
  arXiv  Detail & Related papers  (2026-06-05T17:03:19Z)
• Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization [17.34147279018477]
  We propose a principled framework that combines MAB with cheap predicted scores without compromising statistical validity.<n>We derive doubly robust estimators of each model's performance that use the low-rank predictions to reduce variance.<n> Empirical results on real-world benchmarks show that our approach reduces the number of required evaluations, yielding meaningful savings in compute and cost.
  arXiv  Detail & Related papers  (2026-05-11T11:43:28Z)
• Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization [12.964518425036168]
  We introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences.<n>We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual.<n>Our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines.
  arXiv  Detail & Related papers  (2026-04-08T17:55:00Z)
• Learning Ordinal Probabilistic Reward from Preferences [25.069054134899744]
  We introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM)<n>Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response.<n>Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT)
  arXiv  Detail & Related papers  (2026-02-13T06:43:02Z)
• K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judge [51.93484138861584]
  The rapid development of visual generative models raises the need for more scalable and human-aligned evaluation methods.<n>We propose K-Sort Eval, a reliable and efficient VLM-based evaluation framework that integrates posterior correction and dynamic matching.<n>Experiments show that K-Sort Eval delivers evaluation results consistent with K-Sort Arena, typically requiring fewer than 90 model runs.
  arXiv  Detail & Related papers  (2026-02-10T05:07:46Z)
• Self-Taught Evaluators [77.92610887220594]
  We present an approach that aims to im-proves without human annotations, using synthetic training data only. Our Self-Taught Evaluator can improve a strong LLM from 75.4 to 88.3 on RewardBench.
  arXiv  Detail & Related papers  (2024-08-05T17:57:02Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• On the model-based stochastic value gradient for continuous reinforcement learning [50.085645237597056]
  We show that simple model-based agents can outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. Our findings suggest that model-based policy evaluation deserves closer attention.
  arXiv  Detail & Related papers  (2020-08-28T17:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.