Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

• URL: http://arxiv.org/abs/2608.14425v1
• Date: Fri, 14 Aug 2026 16:06:41 GMT
• Title: Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations
• Abstract summary: optstop treats evaluation as a sequential measurement problem.<n>It keeps every benchmark item eligible for sampling, without requiring a calibrated item bank.<n>It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.

Related papers

• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes [18.54239835812274]
  We propose a Certifiable framework for Efficient LLM evaluation, which leverages E-processes to build anytime-valid CIs.<n>We prove that such signals remain unbiased for the evaluation score conditional on the past, enabling statistically-grounded and anytime-valid CIs.<n>Experiments show that Celeus reaches the target precision using 54-62% fewer evaluated samples than baselines, while preserving anytime-valid coverage.
  arXiv  Detail & Related papers  (2026-06-18T18:07:42Z)
• Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking [40.44372785257615]
  We study inference for this procedure-level target under explicit tuning budgets.<n>We propose SIREN, a selection-aware repeated-split reporting protocol.<n>Controlled simulations and MMLU-Pro tuning experiments show that winner-based reporting can be optimistic.
  arXiv  Detail & Related papers  (2026-05-07T10:18:56Z)
• Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuning [6.908972852063454]
  Uncertainty reliability can degrade earlier and more sharply than top-1 performance.<n>We study this empirically by measuring conformal coverage and calibration error on sequentially fine-tuned models.<n>We propose calibration replay, a lightweight post-hoc procedure that maintains a task-specific buffer and refits a task-specific conformal threshold.
  arXiv  Detail & Related papers  (2026-04-27T03:03:38Z)
• Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores [25.638175689769934]
  We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge)<n>We introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items as possible.<n>Our method uses 2% of the items while improving ranking correlation by 0.12 over random sampling, with 95% accuracy on confident predictions.
  arXiv  Detail & Related papers  (2026-01-20T11:59:13Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding [61.86327960322782]
  We propose a principled framework for randomized decision-making based on interval estimates of the quality of each item.<n>We introduce MERIT, an optimization-based method that maximizes the worst-case expected number of top candidates selected.<n>We prove that MERIT satisfies desirable axiomatic properties not guaranteed by existing approaches.
  arXiv  Detail & Related papers  (2025-06-23T19:59:30Z)
• Towards Reproducible LLM Evaluation: Quantifying Uncertainty in LLM Benchmark Scores [2.886479348067378]
  We use benchmarks designed for testing large language models' capacity to reason about cardinal directions.<n>We suggest a simple method for cost-effectively quantifying the uncertainty of a benchmark score.
  arXiv  Detail & Related papers  (2024-10-04T15:04:28Z)
• Mitigating LLM Hallucinations via Conformal Abstention [70.83870602967625]
  We develop a principled procedure for determining when a large language model should abstain from responding in a general domain. We leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate) Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets.
  arXiv  Detail & Related papers  (2024-04-04T11:32:03Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.