Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification

• URL: http://arxiv.org/abs/2605.28500v1
• Date: Wed, 27 May 2026 14:01:30 GMT
• Title: Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
• Abstract summary: Uncertainty quantification (UQ) methods have emerged as a promising approach for detecting hallucinations in natural language generation.<n>We systematically evaluate how UQ techniques transfer to code generation across three programming languages and over 1,700 problems.<n>We introduce functional equivalence methods, a family of code-specific methods that replace NLI-based semantic equivalence with an LLM-based functional equivalence assessment.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models have shown impressive capabilities in code generation, yet they often produce functionally incorrect code. Uncertainty quantification (UQ) methods have emerged as a promising approach for detecting hallucinations in natural language generation, but their effectiveness for code generation tasks remains underexplored. We systematically evaluate how UQ techniques transfer to code generation across three programming languages, five LLMs, and over 1,700 problems. We find that some token-probability-based methods generalize effectively without modification, while sampling-based methods relying on natural language inference (NLI) fail because NLI models cannot distinguish functionally different code, causing most responses to collapse into a single semantic cluster. To address this, we introduce functional equivalence methods, a family of code-specific methods that replace NLI-based semantic equivalence with an LLM-based functional equivalence assessment, including functional entropy, a code-specific analog of semantic entropy. Functional equivalence methods achieve top AUROC in 11 out of 15 model-benchmark combinations and the best calibration across most settings, consistently outperforming both NLI-based counterparts and all other methods evaluated.

Related papers

• Evaluating Language Models on Cross-Language Code Functional Equivalence [6.76025470599609]
  We introduce PolyHuman, a dataset of human-written programs in CPP, Java, and Python.<n>Using this dataset, we evaluate intra- and inter-language equivalence detection across open-weight and proprietary LLMs.<n>We analyze 81 cases of systematic disagreement in which models incorrectly judge functional equivalence.
  arXiv  Detail & Related papers  (2026-08-25T01:41:00Z)
• ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code Generation [27.9837392531619]
  We propose a knowledge-infused framework named ShortCoder to optimize code generation efficiency.<n>ShortCoder consistently outperforms state-of-the-art methods on HumanEval.
  arXiv  Detail & Related papers  (2026-01-14T18:57:31Z)
• Dynamic Stability of LLM-Generated Code [6.120340803716395]
  Current evaluations of LLMs for code generation overlook the fact that functionally correct solutions can differ significantly in algorithmic complexity.<n>We introduce a principled framework for evaluating the dynamic stability of generated code.<n>Our findings call for stability-aware objectives in code generation and new benchmarks with test cases for robust, real-world evaluation.
  arXiv  Detail & Related papers  (2025-11-07T09:58:06Z)
• HyClone: Bridging LLM Understanding and Dynamic Execution for Semantic Code Clone Detection [3.2167919219391474]
  Code clone detection is a critical task in software engineering, aimed at identifying duplicated or similar code fragments within or across software systems.<n>Recent advances in large language models (LLMs) have shown promise in understanding code semantics.<n>We propose a novel two-stage framework that combines LLM-based screening with execution-based validation for detecting semantic clones in Python programs.
  arXiv  Detail & Related papers  (2025-08-02T13:11:56Z)
• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• Function-to-Style Guidance of LLMs for Code Translation [59.487054943812836]
  We propose F2STrans, a function-to-style guiding paradigm designed to improve the performance of large language models in code translation.<n>Our approach comprises two key stages: (1) Functional learning, which optimize translation correctness using high-quality source-target code pairs.<n>We introduce a novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth translations.
  arXiv  Detail & Related papers  (2025-07-15T08:25:02Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models [76.17975723711886]
  Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs)<n>In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation.<n>Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores.
  arXiv  Detail & Related papers  (2025-02-20T10:25:13Z)
• The First Prompt Counts the Most! An Evaluation of Large Language Models on Iterative Example-Based Code Generation [33.77058239791512]
  This paper presents the first comprehensive study on example-based code generation using Large Language Models (LLMs)<n>We adopt an iterative evaluation framework and formalize the objective of example-based code generation as two sequential sub-objectives.<n>We assess six state-of-the-art LLMs using a new benchmark of 172 diverse target functionalities.
  arXiv  Detail & Related papers  (2024-11-11T08:05:37Z)
• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification is a key element of machine learning applications.<n>We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines.<n>We conduct a large-scale empirical investigation of UQ and normalization techniques across eleven tasks, identifying the most effective approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.