Ensemble-Based Uncertainty Estimation for Code Correctness Estimation

• URL: http://arxiv.org/abs/2603.27098v2
• Date: Sat, 04 Apr 2026 02:18:57 GMT
• Title: Ensemble-Based Uncertainty Estimation for Code Correctness Estimation
• Abstract summary: We propose Ensemble Semantic Entropy (ESE), which estimates uncertainty by evaluating the consistency of samples aggregated across an ensemble of models.<n>Experiments on LiveCodeBench demonstrate that ESE correlates more strongly with program correctness than single-model semantic entropy.<n>We also propose a cascading test-time scaling framework Cas, which maintains performance while reducing FLOPs by 64.9% compared to single-model scaling.
• Score: 36.53771380869671
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in generating programs from natural language descriptions, yet ensuring their correctness without an external oracle remains a critical challenge. To solve the challenge, existing methods often rely on uncertainty estimation, measuring the consistency of semantics or execution behaviors across multiple samples generated by a single model. However, we observe that a single model can often converge to a consistent but incorrect solution, rendering such consistency-based proxies ineffective. To address this, we propose Ensemble Semantic Entropy (ESE), which estimates uncertainty by evaluating the consistency of samples aggregated across an ensemble of models. Experiments on LiveCodeBench demonstrate that ESE correlates more strongly with program correctness than single-model semantic entropy. Notably, in selective generation tasks with strict false-positive rate constraints, ESE improves prediction accuracy by 53.4%. Furthermore, by leveraging ESE as the decision signal, we propose a cascading test-time scaling framework Cas, which maintains performance while reducing FLOPs by 64.9% compared to single-model scaling, offering a new perspective on balancing parameter and inference scaling.

Related papers

• Calibration-Aware Uncertainty Cascades for Efficient Heterogeneous Model Collaboration [18.045450334322545]
  Heterogeneous model collaboration seeks to exploit the complementary strengths of different models to balance predictive performance and inference cost.<n>Existing approaches typically rely on trained routers, which tie routing decisions to a fixed task and model pool, or on raw-confidence cascades, whose thresholds lack consistent reliability semantics across heterogeneous models.<n>We propose-Aware Uncertainty Cascades (CAUC), a simple post-hoc framework that independently calibrates each model's confidence and selects deployment policies using validation data.
  arXiv  Detail & Related papers  (2026-09-10T12:14:54Z)
• Statistical Inference for Misspecified Contextual Bandits [6.178061357164435]
  We show a previously overlooked issue: standard algorithms may fail to stabilize under misspecified working models.<n>We develop an inverse-probability-weighted Z-estimation framework for a broad class of marginal moment targets.<n>We establish sufficient conditions for scaled inverse-propensity convergence for several policy classes.
  arXiv  Detail & Related papers  (2026-06-21T18:57:13Z)
• Clustered Self-Assessment: A Simple yet Effective Method for Uncertainty Quantification in Large Language Models [44.59799077650502]
  We propose a simple yet effective self-assessment method for uncertainty quantification in large language models.<n>Our approach groups sampled generations into semantically distinct clusters, converts them into answer options in a structured multiple-choice question, and uses the probability assigned by the model to each option as a confidence estimate.
  arXiv  Detail & Related papers  (2026-06-02T16:25:54Z)
• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• Symmetric Aggregation of Conformity Scores for Efficient Uncertainty Sets [6.673032375204486]
  We propose SACP (Symmetric Aggregated Conformal Prediction), a novel method that aggregates nonconformity scores from multiple predictors.<n>SACP transforms these scores into e-values and combines them using any symmetric aggregation function.<n>We show that SACP consistently improves efficiency and often outperforms state-of-the-art model aggregation baselines.
  arXiv  Detail & Related papers  (2025-12-07T17:54:07Z)
• Penalized Empirical Likelihood for Doubly Robust Causal Inference under Contamination in High Dimensions [0.720409153108429]
  We propose a doubly robust estimator for the average treatment effect in low sample size equations.<n>We show that the proposed confidence interval remain efficient compared to those competing estimates.
  arXiv  Detail & Related papers  (2025-07-23T11:58:54Z)
• Accelerated Test-Time Scaling with Model-Free Speculative Sampling [58.69141724095398]
  We introduce STAND (STochastic Adaptive N-gram Drafting), a novel model-free speculative decoding approach.<n>We show that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding.<n>As a model-free approach, STAND can be applied to any existing language model without additional training.
  arXiv  Detail & Related papers  (2025-06-05T07:31:18Z)
• Causal Invariance Learning via Efficient Optimization of a Nonconvex Objective [12.423111378195667]
  We propose a novel method for finding causal relationships among variables.<n>We show that our method converges to a standard gradient method.<n>Our algorithm avoids exhaustive searches, making it especially when the number of covariants is large.
  arXiv  Detail & Related papers  (2024-12-16T15:11:02Z)
• Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods [59.779795063072655]
  Chain-of-Thought (CoT) prompting and its variants have gained popularity as effective methods for solving multi-step reasoning problems. We analyze CoT prompting from a statistical estimation perspective, providing a comprehensive characterization of its sample complexity.
  arXiv  Detail & Related papers  (2024-08-25T04:07:18Z)
• Evaluating Model Performance Under Worst-case Subpopulations [8.615300901890253]
  We study the worst-case performance of a model over all subpopulations of a given size, defined with respect to core attributes Z. We develop a scalable yet principled two-stage estimation procedure that can evaluate the robustness of state-of-the-art models.
  arXiv  Detail & Related papers  (2024-07-01T14:24:05Z)
• The Implicit Delta Method [61.36121543728134]
  In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of uncertainty. We show that the change in the evaluation due to regularization is consistent for the variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference.
  arXiv  Detail & Related papers  (2022-11-11T19:34:17Z)
• Error-based Knockoffs Inference for Controlled Feature Selection [49.99321384855201]
  We propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-03-09T01:55:59Z)
• Calibrating Over-Parametrized Simulation Models: A Framework via Eligibility Set [3.862247454265944]
  We develop a framework to develop calibration schemes that satisfy rigorous frequentist statistical guarantees. We demonstrate our methodology on several numerical examples, including an application to calibration of a limit order book market simulator.
  arXiv  Detail & Related papers  (2021-05-27T00:59:29Z)
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation [74.06904875527556]
  We propose a generic and efficient segmentation framework to construct ensemble segmentation models. In the proposed method, ensemble models can be efficiently generated by using the layer selection method. We also devise a new pixel-wise uncertainty loss, which improves the predictive performance.
  arXiv  Detail & Related papers  (2020-05-21T16:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.