Code Is More Than Text: Uncertainty Estimation for Code Generation

• URL: http://arxiv.org/abs/2606.09577v1
• Date: Mon, 08 Jun 2026 14:52:43 GMT
• Title: Code Is More Than Text: Uncertainty Estimation for Code Generation
• Abstract summary: We argue that code differs from natural language (NL) in three ways: a single wrong token can break an entire program (token), algorithmic intent and concrete implementation can disagree independently (intent-code gap), and programs can be executed (executability)<n>We instantiate these properties as three uncertainty axes: lexical (Top-K token entropy), algorithmic fragility (pseudo-code consistency), and functional (behavioral consistency)
• Score: 28.04063266984485
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE) is essential for selective prediction, human-in-the-loop review, and downstream agentic decisions. Yet most existing code UE methods are inherited from natural language (NL) generation and ignore properties that make code distinct. We argue that code differs from NL in three ways: a single wrong token can break an entire program (token fragility); algorithmic intent and concrete implementation can disagree independently (intent-code gap); and programs can be executed (executability). We instantiate these properties as three orthogonal uncertainty axes: lexical (Top-K token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency). Across five code LLMs, our three-axis ensemble improves average AUROC from 0.696 for the strongest NL-derived baseline to 0.776 (+8.1 points). Notably, on Qwen3-14B, our single-pass Top-K token entropy matches the strongest multi-pass baseline while being over 3x cheaper; across models, it remains a competitive low-cost signal. These results suggest that code UE deserves code-specific design rather than direct NL ports.

Related papers

• When Uncertainty Isn't Enough: An Empirical Study of Self-Correction in Code Generation [3.657882835504143]
  We study whether uncertainty estimation methods developed for natural language transfer to code generation, and whether such signals can improve code generation via selective self-correction.<n>We evaluate five uncertainty methods: mean token entropy, verbalized confidence, $P(textTrue)$, entropy ensembles, and semantic entropy probes.<n>We find that multi-sample $P(textTrue)$ achieves the strongest correlation with correctness, while all the other methods, including semantic entropy probes, yield only weak correlation.
  arXiv  Detail & Related papers  (2026-07-31T21:13:02Z)
• NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming [62.89531126732269]
  We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation.<n>We construct a semantically validated pool of 23,594 examples, and a slot-balanced 800 diagnostic benchmark.<n>Experiments across instruction-tuned and fine-tuned LLMs reveal a notable gap between lexical similarity and semantic alignment.
  arXiv  Detail & Related papers  (2026-06-20T14:22:59Z)
• Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMs [12.164962289028113]
  We introduce Tree-like Self-Play (TSP), a framework that reframes secure code generation as a fine-grained sequential decision process.<n>By treating code generation as a self-play game, the model learns to strictly discriminate against its own localized errors.
  arXiv  Detail & Related papers  (2026-06-02T11:07:20Z)
• Inferring Code Correctness from Specification [0.0]
  Large language models (LLMs) have become integral to modern software development, enabling automated code generation at scale.<n>We propose TRAILS (Targeted Reasoning Agreement via Inputs and Specifications), an approach that grounds LLM reasoning with concrete (input, output) pairs.<n>We evaluate TRAILS on two datasets, LiveCodeBench and CoCoClaNeL, across three LLMs (Qwen3Coder-30B, Devstral-Small-24B, and Olmo3.1-Instruct), comparing against HoarePrompt and a Zero-Shot Chain-of-Thought baseline.
  arXiv  Detail & Related papers  (2026-05-28T12:04:51Z)
• Reasoning, Code, or Both? How Large Language Models Handle Variations in Math Questions [0.0]
  This study evaluates three approaches on 1,000 problems from the GSM-Symbolic dataset.<n>All three were run on paired original and modified problems using Claude Haiku 4.5.<n>CoT was the most robust method, with an accuracy drop of 1.3 percentage points and 1.8% of problems breaking under perturbation.
  arXiv  Detail & Related papers  (2026-05-26T00:44:49Z)
• Secure Code Generation at Scale with Reflexion [0.0]
  Python yields the highest secure rates; C and C# are the lowest.<n>Reflexion prompting improves security for all models, improving average accuracy from 70.74% at t0 to 79.43% at t3.
  arXiv  Detail & Related papers  (2025-11-05T22:46:24Z)
• Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection [0.0]
  Three industry-standard rule-based static code-analysis tools (Sonar, CodeQL and Snyk Code) and three state-of-the-art large language models hosted on the GitHub Models platform (GPT-4.1, Mistral Large and DeepSeek V3) were evaluated.<n>Using a curated suite of ten real-world C# projects that embed 63 vulnerabilities, we measure classical accuracy (precision, recall, F-score), analysis latency, granularity and the developer effort required to vet true positives.<n>We recommend a hybrid pipeline: employ language models early in development for broad, context-aware detection and
  arXiv  Detail & Related papers  (2025-08-06T13:48:38Z)
• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• $\mathbb{USCD}$: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decoding [64.00025564372095]
  Large language models (LLMs) have shown remarkable capabilities in code generation. The effects of hallucinations (e.g., output noise) make it challenging for LLMs to generate high-quality code in one pass. We propose a simple and effective textbfuncertainty-aware textbfselective textbfcontrastive textbfdecoding.
  arXiv  Detail & Related papers  (2024-09-09T02:07:41Z)
• Decoding at the Speed of Thought: Harnessing Parallel Decoding of Lexical Units for LLMs [57.27982780697922]
  Large language models have demonstrated exceptional capability in natural language understanding and generation. However, their generation speed is limited by the inherently sequential nature of their decoding process. This paper introduces Lexical Unit Decoding, a novel decoding methodology implemented in a data-driven manner.
  arXiv  Detail & Related papers  (2024-05-24T04:35:13Z)
• CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code [75.08995072899594]
  We propose CodeBERTScore: an evaluation metric for code generation. CodeBERTScore encodes the natural language input preceding the generated code. We find that CodeBERTScore achieves a higher correlation with human preference and with functional correctness than all existing metrics.
  arXiv  Detail & Related papers  (2023-02-10T22:12:05Z)
• Execution-Based Evaluation for Open-Domain Code Generation [81.96731162394445]
  ODEX is the first Open-Domain EXecution-based natural language (NL) to Python code generation dataset. ODEX has 945 NL-Code pairs spanning 79 diverse libraries, along with 1,707 human-written test cases for execution. ODEX supports four natural languages as intents, in English, Spanish, Japanese, and Russian.
  arXiv  Detail & Related papers  (2022-12-20T17:54:37Z)
• Interactive Code Generation via Test-Driven User-Intent Formalization [60.90035204567797]
  Large language models (LLMs) produce code from informal natural language (NL) intent. It is hard to define a notion of correctness since natural language can be ambiguous and lacks a formal semantics. We describe a language-agnostic abstract algorithm and a concrete implementation TiCoder.
  arXiv  Detail & Related papers  (2022-08-11T17:41:08Z)
• Natural Language to Code Translation with Execution [82.52142893010563]
  Execution result--minimum Bayes risk decoding for program selection. We show that it improves the few-shot performance of pretrained code models on natural-language-to-code tasks.
  arXiv  Detail & Related papers  (2022-04-25T06:06:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.