When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation

• URL: http://arxiv.org/abs/2608.14659v1
• Date: Fri, 31 Jul 2026 21:13:02 GMT
• Title: When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation
• Abstract summary: We study whether uncertainty estimation methods developed for natural language transfer to code generation, and whether such signals can improve code generation via selective self-correction.<n>We evaluate five uncertainty methods: mean token entropy, verbalized confidence, $P(textTrue)$, entropy ensembles, and semantic entropy probes.<n>We find that multi-sample $P(textTrue)$ achieves the strongest correlation with correctness, while all the other methods, including semantic entropy probes, yield only weak correlation.
• Score: 3.657882835504143
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models for code generation often produce incorrect solutions without reliable indicators of failure. We study whether uncertainty estimation methods developed for natural language transfer to code generation, and whether such signals can improve code generation via selective self-correction. We evaluate five uncertainty methods: mean token entropy, verbalized confidence, $P(\text{True})$, entropy ensembles, and semantic entropy probes, across three small code LLMs on HumanEval and BigCodeBench. We find that multi-sample $P(\text{True})$ achieves the strongest correlation with correctness, while all the other methods, including semantic entropy probes, yield only weak correlation. We then use these uncertainty signals to drive three self-correction policies: adaptive decoding, uncertainty-based regeneration, and verification-based regeneration. Our results reveal a stronger negative finding than anticipated: uncertainty-based self-correction fails to reliably improve Pass@1, degrading accuracy in 5 of 6 configurations across both benchmarks ($-3$pp to $-10$pp), and adaptive decoding degrades accuracy in 4 of 6 configurations. Only verification-based self-correction reliably improves Pass@1, with gains of $+6$ to $+26$ percentage points on HumanEval and $+8$ to $+20$ percentage points on BigCodeBench, scaling inversely with baseline strength. These findings replicate consistently across both benchmarks and suggest that cheap uncertainty estimators are insufficient on their own to improve code correctness, and that their practical value lies in serving as gating signals for costlier execution-based correction loops rather than as standalone substitutes for verification.

Related papers

• $TCP_α$: Margin-Controlled Confidence estimation for reliable Music Information Retrieval [7.679660052685777]
  Deep neural networks are often overconfident, assigning high confidence even to incorrect predictions.<n>Existing targets suffer from inherent ambiguity: they assign overlapping confidence values to correct and incorrect predictions, while errors near the decision boundary receive confidence scores indistinguishable from correct predictions.<n>We propose $ TCP_$, a novel confidence target that resolves these limitations by introducing a margin-controlled penalty for misclassified samples.
  arXiv  Detail & Related papers  (2026-08-20T17:58:50Z)
• GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning [61.67411833252235]
  We introduce GradCuit, which inserts optimizable latent states at a selected Transformer layer between the hidden representations of the prompt and the generated continuation.<n>Across five instruction-tuned backbones, three reasoning benchmarks, and two answer formats, GradCuit achieves an average accuracy of 64.5%.<n>For interpretability, token-level gradient attribution reveals that latent influence concentrates on reasoning-connector tokens, while layer analysis identifies early-to-middle Transformer layers as the most effective optimization space.
  arXiv  Detail & Related papers  (2026-08-03T17:55:24Z)
• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• Do Uncertainty Signals Help? A Systematic Study of Uncertainty-Aware Decoding with Rollback Mechanisms [21.868021164923515]
  We study an inference-time strategy that uses uncertainty signals to identify unreliable generation regions and roll back to earlier valid prefixes without retraining the model.<n>Our results show that the complete rollback framework improves over equal-budget restart across the evaluated benchmarks and model settings.
  arXiv  Detail & Related papers  (2026-07-31T05:26:42Z)
• SVR: Self-Verifying Refinement via Joint Verdict-Confidence Reinforcement Learning for Adaptive Test-Time Compute [62.3458279176813]
  Self-Verifying Refinement is an oracle-free multi-turn reinforcement learning framework.<n>It learns to use self-verification as a compute-control policy.<n>It achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
  arXiv  Detail & Related papers  (2026-07-30T16:20:58Z)
• Pause and Reflect: Conformal Aggregation for Chain-of-Thought Reasoning [8.024041325202612]
  Chain-of-thought (CoT) reasoning with self-consistency improves performance by aggregating multiple sampled reasoning paths.<n>We introduce a conformal procedure for CoT reasoning that directly addresses aggregation uncertainty.<n>Our approach replaces majority voting with weighted score aggregation over reasoning paths and calibrates an abstention rule using conformal risk control.
  arXiv  Detail & Related papers  (2026-05-13T20:33:59Z)
• Reducing Hallucinations in LLM-Generated Code via Semantic Triangulation [2.8646222242803643]
  We introduce semantic triangulation, which transforms a programming problem in a way that preserves an exact, verifiable mapping between solutions.<n>On the LiveCodeBench and CodeElo benchmarks, semantic triangulation increases reliability of generated code by 21%.<n>It is also the only approach to consistently form true consensus on tasks with multiple valid but non-equivalent solutions.
  arXiv  Detail & Related papers  (2025-11-15T16:45:14Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• Towards Better Code Generation: Adaptive Decoding with Uncertainty Guidance [42.737012213197865]
  AdaDec is an adaptive decoding framework that employs a lookahead-based, uncertainty-aware pause-and-rerank mechanism.<n>AdaDec achieves up to 20.9% absolute gains in Pass@1 accuracy compared with greedy decoding.<n>By applying reranking only when necessary, AdaDec reduces computational overhead and latency, enhancing efficiency alongside reliability.
  arXiv  Detail & Related papers  (2025-06-10T16:49:46Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• Self-Evaluation Guided Beam Search for Reasoning [61.523627290397556]
  We introduce a stepwise self-evaluation mechanism to guide and calibrate the reasoning process of Large Language Model (LLM) We propose a decoding algorithm integrating the self-evaluation guidance via beam search. Our approach surpasses the corresponding Codex-backboned baselines in few-shot accuracy by $6.34%$, $9.56%$, and $5.46%$ on the GSM8K, AQuA, and StrategyQA.
  arXiv  Detail & Related papers  (2023-05-01T02:37:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.