Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals

• URL: http://arxiv.org/abs/2602.03061v1
• Date: Tue, 03 Feb 2026 03:40:01 GMT
• Title: Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals
• Authors: Zihan Dong, Zhixian Zhang, Yang Zhou, Can Jin, Ruijia Wu, Linjun Zhang,
• Abstract summary: We develop a framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains.<n>Across simulations, our one-step estimator substantially improves ranking accuracy with gains increasing as model output noise grows.<n>Experiments on GPQA Diamond, AIME 2025 and GSM8K further demonstrate more precise performance estimation and more reliable model rankings.
• Score: 18.612081365101464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating mathematical reasoning in LLMs is constrained by limited benchmark sizes and inherent model stochasticity, yielding high-variance accuracy estimates and unstable rankings across platforms. On difficult problems, an LLM may fail to produce a correct final answer, yet still provide reliable pairwise comparison signals indicating which of two candidate solutions is better. We leverage this observation to design a statistically efficient evaluation framework that combines standard labeled outcomes with pairwise comparison signals obtained by having models judge auxiliary reasoning chains. Treating these comparison signals as control variates, we develop a semiparametric estimator based on the efficient influence function (EIF) for the setting where auxiliary reasoning chains are observed. This yields a one-step estimator that achieves the semiparametric efficiency bound, guarantees strict variance reduction over naive sample averaging, and admits asymptotic normality for principled uncertainty quantification. Across simulations, our one-step estimator substantially improves ranking accuracy, with gains increasing as model output noise grows. Experiments on GPQA Diamond, AIME 2025, and GSM8K further demonstrate more precise performance estimation and more reliable model rankings, especially in small-sample regimes where conventional evaluation is pretty unstable.

Related papers

• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Efficient Inference for Noisy LLM-as-a-Judge Evaluation [8.2511120576505]
  Large language models (LLMs) are increasingly used as automatic evaluators of generative AI outputs.<n>In practice, LLM judges are imperfect predictions for the underlying truth and can exhibit systematic, non-random errors.
  arXiv  Detail & Related papers  (2026-01-08T22:46:26Z)
• Bayesian Semiparametric Causal Inference: Targeted Doubly Robust Estimation of Treatment Effects [1.2833734915643464]
  We propose a semiparametric Bayesian methodology for estimating the average treatment effect (ATE) within the potential outcomes framework.<n>Our method introduces a Bayesian debiasing procedure that corrects for bias arising from nuisance estimation.<n>Extensive simulations confirm the theoretical results, demonstrating accurate point estimation and credible intervals with nominal coverage.
  arXiv  Detail & Related papers  (2025-11-19T22:15:04Z)
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics [72.00014675808228]
  Instability in Large Language Models evaluation process obscures true learning dynamics.<n>We introduce textbfMaP, a framework that integrates underlineMerging underlineand the underlinePass@k metric.<n>Experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent rankings.
  arXiv  Detail & Related papers  (2025-10-10T11:40:27Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Improving Value-based Process Verifier via Low-Cost Variance Reduction [24.609940184050043]
  Large language models (LLMs) have achieved remarkable success in a wide range of tasks.<n>However, their reasoning capabilities, particularly in complex domains like mathematics, remain a significant challenge.<n>Value-based process verifiers, which estimate the probability of a partial reasoning chain leading to a correct solution, are a promising approach for improving reasoning.
  arXiv  Detail & Related papers  (2025-08-14T11:22:29Z)
• Spectral Ranking Inferences based on General Multiway Comparisons [7.222667862159246]
  We show that a two-step spectral method can achieve the same vanilla efficiency as the Maximum Likelihood Estor. It is noteworthy that this is the first time effective two-sample rank testing methods have been proposed.
  arXiv  Detail & Related papers  (2023-08-05T16:31:32Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• Federated Edge Learning with Misaligned Over-The-Air Computation [36.39188653838991]
  Over-the-air computation (OAC) is a promising technique to realize fast model aggregation in the uplink of federated edge learning. How to design the maximum likelihood (ML) estimator in the presence of residual channel-gain mismatch and asynchronies is an open problem. This paper formulates the problem of misaligned OAC for federated edge learning and puts forth a whitened matched filtering and sampling scheme.
  arXiv  Detail & Related papers  (2021-02-26T17:19:56Z)
• Instability, Computational Efficiency and Statistical Accuracy [101.32305022521024]
  We develop a framework that yields statistical accuracy based on interplay between the deterministic convergence rate of the algorithm at the population level, and its degree of (instability) when applied to an empirical object based on $n$ samples. We provide applications of our general results to several concrete classes of models, including Gaussian mixture estimation, non-linear regression models, and informative non-response models.
  arXiv  Detail & Related papers  (2020-05-22T22:30:52Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.