Do Repetitions Matter? Strengthening Reliability in LLM Evaluations

• URL: http://arxiv.org/abs/2509.24086v1
• Date: Sun, 28 Sep 2025 21:45:20 GMT
• Title: Do Repetitions Matter? Strengthening Reliability in LLM Evaluations
• Abstract summary: LLM leaderboards often rely on single runs, but how many repetitions are required for reliable conclusions remains unclear.<n>We re-evaluated eight state-of-the-art models on the AI4Math Benchmark with three independent runs per setting.
• Score: 0.374750127323442
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: LLM leaderboards often rely on single stochastic runs, but how many repetitions are required for reliable conclusions remains unclear. We re-evaluate eight state-of-the-art models on the AI4Math Benchmark with three independent runs per setting. Using mixed-effects logistic regression, domain-level marginal means, rank-instability analysis, and run-to-run reliability, we assessed the value of additional repetitions. Our findings shows that Single-run leaderboards are brittle: 10/12 slices (83\%) invert at least one pairwise rank relative to the three-run majority, despite a zero sign-flip rate for pairwise significance and moderate overall interclass correlation. Averaging runs yields modest SE shrinkage ($\sim$5\% from one to three) but large ranking gains; two runs remove $\sim$83\% of single-run inversions. We provide cost-aware guidance for practitioners: treat evaluation as an experiment, report uncertainty, and use $\geq 2$ repetitions under stochastic decoding. These practices improve robustness while remaining feasible for small teams and help align model comparisons with real-world reliability.

Related papers

• Recursive Agentic Reasoning [4.7490116928645065]
  Test-time reasoning methods such as iterative refinement, decomposition, and repeated sampling are often evaluated in isolation.<n>We introduce a unified view of these methods as recursion operators over an agent's reasoning trace.<n>We evaluate all three operators against a single-pass chain-of-thought baseline under a shared harness with identical prompts, token budgets, and grading code.
  arXiv  Detail & Related papers  (2026-08-25T01:34:08Z)
• enhancing reasoning accuracy in large language models during inference time [1.0282918759603745]
  Large Language Models (LLMs) often exhibit strong linguistic abilities while remaining unreliable on multi-step reasoning tasks.<n>We study inference-time techniques to improve the reasoning accuracy of LLMs.
  arXiv  Detail & Related papers  (2026-03-22T16:00:07Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation [3.23600523782706]
  We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multimodal reasoning.<n>We propose two complementary metrics: Match F1, which scores step-level precision and recall via semantic similarity matching, and Ordered Match F1, which penalizes disordered reasoning chains.<n>CPR-Curriculum achieves a 32% improvement in Match F1 via GRPO where additive reward strategies fail, improving reasoning without manual step annotation.
  arXiv  Detail & Related papers  (2026-03-13T15:48:15Z)
• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• Beyond Correctness: Learning Robust Reasoning via Transfer [51.403609251508904]
  We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it.<n>We introduce Reinforcement Learning with Transferable Reward, which operationalizes robustness via transfer reward.<n>Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps.
  arXiv  Detail & Related papers  (2026-02-09T10:41:44Z)
• MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning [85.05204262206296]
  Large reasoning models (LRMs) achieve strong performance by producing long chains of thought, but their inference costs are high.<n>Small language models (SLMs) are far more efficient, yet struggle on multi-step reasoning tasks.<n>We propose MentorCollab, an inference-time collaboration method in which an LRM selectively and sparsely guides an SLM, rather than taking over generation.
  arXiv  Detail & Related papers  (2026-02-05T04:58:16Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs [2.125148574616104]
  Large Language Models (LLMs) often produce inconsistent answers when faced with different phrasings of the same prompt.<n>We propose Flip-Flop Consistency ($F2C$), an unsupervised training method that improves robustness to such perturbations.<n>We evaluate our method on 11 datasets spanning four NLP tasks, with 4-15 prompt variations per dataset.
  arXiv  Detail & Related papers  (2025-10-16T02:54:01Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning [0.5204229323525671]
  We present a counterfactual reward model that introduces causal inference with multimodal representation learning to provide an unsupervised, bias-resilient reward signal.<n>We evaluated the framework on a multimodal fake versus true news dataset, which exhibits framing bias, class imbalance, and distributional drift.<n>The resulting system achieved an accuracy of 89.12% in fake news detection, outperforming the baseline reward models.
  arXiv  Detail & Related papers  (2025-08-27T04:54:33Z)
• Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision [20.09181711927194]
  We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM.<n>On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 times less data.
  arXiv  Detail & Related papers  (2025-05-26T08:56:36Z)
• Classifying Inconsistency in AHP Pairwise Comparison Matrices Using Machine Learning [0.0]
  This study introduces a novel alternative that leverages triadic preference reversals (PR) to provide a more robust and interpretable assessment of consistency.<n>The PR method achieves 97% accuracy, significantly surpassing the Consistency Ratio (CR) method's 50%, with a false negative rate of only 2.6% compared to 5.5%.
  arXiv  Detail & Related papers  (2025-05-07T14:17:39Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step.<n>Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy.<n>We propose Reasoning-Driven Process Reward Modeling (R-PRM)<n>R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning [80.15393178083607]
  Large Language Models' (LLM) reasoning can be improved using test-time aggregation strategies, i.e., generating multiple samples and voting among generated samples.<n> Refinement offers an alternative by using LLM-generated feedback to improve solution quality.<n>We propose MAgICoRe, which avoids excessive refinement by categorizing problem difficulty as easy or hard.
  arXiv  Detail & Related papers  (2024-09-18T17:12:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.