What Fixed-Rollout pass@k Evaluations Can Identify

• URL: http://arxiv.org/abs/2609.09245v1
• Date: Tue, 08 Sep 2026 10:27:52 GMT
• Title: What Fixed-Rollout pass@k Evaluations Can Identify
• Abstract summary: Repeated-sampling evaluations extrapolate pass@k far beyond the number n of samples collected per problem.<n>We show that, in the pooled/random-task conditional-Binomial model, fixed-n success counts identify only the n free moments of the latent per-task success distribution.
• Score: 3.0318216701273397
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/random-task conditional-Binomial model, fixed-n success counts identify only the n free moments of the latent per-task success distribution. Consequently, direct pass@k is identified for k <= n, but generic extrapolated pass@k, tail exponents, and tail constants are not identified for k > n, even with arbitrarily many exchangeable tasks at the same rollout budget. This is stronger than the observation that the usual estimator is undefined beyond n: it characterizes the information missing from the fixed-depth count-law experiment. We give exact count-law-preserving constructions with incompatible extrapolations, state the exceptional unique-extension case, and compute sharp population identified intervals through Hausdorff principal representations. On the public 10,000-rollout-per-problem release of Brown et al., counterfactual n = 16 evaluations leave failure at k = 1000 ambiguous by factors from 1.5 to over 2,600 across four MATH/GSM8K/CodeContests configurations. The calibration shows that intermediate-scale failure share alone does not determine width. Our result does not reject parametric inference-time scaling laws; it supplies the nonparametric baseline against which their assumptions can be evaluated. We give an exact, conservative one-coordinate finite-task confidence certificate and a reporting standard separating direct estimates, identified sets, and model-conditioned forecasts.

Related papers

• Optimal Stratified Allocation for Rare-Event Onset Forecasting in Dependent Sequences [2.2632368327435732]
  We study estimation of total risk from a subsample K n under designs allocating K0 and K1 draws to the two strata.<n>We derive the exact finite-population variance of the weighted risk estimator under class-conditional sampling without replacement.
  arXiv  Detail & Related papers  (2026-09-03T19:38:01Z)
• Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition [0.0]
  Conditional Jensen-Shannon Discrepancy (CJSD)<n>On a ten-measure battery over 202 dataset pairs, only the two conditional-information estimators -- CJSD and a kNN plug-in for the same estimand -- fail.
  arXiv  Detail & Related papers  (2026-08-20T10:48:08Z)
• PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Data [0.0]
  We introduce PAIR-CI, a nonparametric conditional independence (CI) test that restores calibration by integrating multiple imputation directly into the inferential procedure.<n>A provably consistent variance estimator jointly accounts for uncertainty arising from cross-validation and multiple imputation.
  arXiv  Detail & Related papers  (2026-05-06T12:34:37Z)
• Online Covariance Estimation in Nonsmooth Stochastic Approximation [14.818683408659764]
  We consider applying approximation (SA) methods to solve nonsmooth variational inclusion problems.<n>Our convergence construction establish the best-known for statistical estimation methods.
  arXiv  Detail & Related papers  (2025-02-07T20:16:51Z)
• Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
  Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs.<n>We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint.<n>We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
  arXiv  Detail & Related papers  (2024-06-05T13:36:38Z)
• Model-Agnostic Covariate-Assisted Inference on Partially Identified Causal Effects [1.9253333342733674]
  Many causal estimands are only partially identifiable since they depend on the unobservable joint distribution between potential outcomes. We propose a unified and model-agnostic inferential approach for a wide class of partially identified estimands.
  arXiv  Detail & Related papers  (2023-10-12T08:17:30Z)
• Nonparametric extensions of randomized response for private confidence sets [51.75485869914048]
  This work derives methods for performing nonparametric, nonasymptotic statistical inference for population means under the constraint of local differential privacy (LDP) We present confidence intervals (CI) and time-uniform confidence sequences (CS) for $mustar$ when only given access to the privatized data.
  arXiv  Detail & Related papers  (2022-02-17T16:04:49Z)
• Universal Off-Policy Evaluation [64.02853483874334]
  We take the first steps towards a universal off-policy estimator (UnO) We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns.
  arXiv  Detail & Related papers  (2021-04-26T18:54:31Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)
• Nonparametric Score Estimators [49.42469547970041]
  Estimating the score from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models. We provide a unifying view of these estimators under the framework of regularized nonparametric regression. We propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.
  arXiv  Detail & Related papers  (2020-05-20T15:01:03Z)
• Estimating Gradients for Discrete Random Variables by Sampling without Replacement [93.09326095997336]
  We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators.
  arXiv  Detail & Related papers  (2020-02-14T14:15:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.