An Empirical Study of Security Calibration in Large Language Models for Code

• URL: http://arxiv.org/abs/2606.31159v1
• Date: Tue, 30 Jun 2026 05:37:39 GMT
• Title: An Empirical Study of Security Calibration in Large Language Models for Code
• Abstract summary: We present the first large-scale empirical study of security calibration in LLM-generated code.<n>We evaluate GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across multiple benchmarks.<n>Our results suggest that overconfidence is prevalent across the evaluated LLMs.
• Score: 5.167904179040144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure? This property, known as calibration, measures whether a model's confidence aligns with the true correctness of its outputs. We present the first large-scale empirical study of security calibration in LLM-generated code. We evaluate GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across multiple temperature settings on two complementary benchmarks: self-contained security tasks and multi-language repository-level contexts. Our results suggest that overconfidence is prevalent across the evaluated LLMs. Functional calibration is consistently worse than security calibration, suggesting that models estimate security outcomes more reliably than functional correctness, potentially because functional correctness depends on complex execution behavior. We also examine whether calibration-guided automated repair can help remediate vulnerabilities in LLM-generated code, finding only limited improvements while frequently introducing functional regressions. Moreover, we study different mitigation strategies for reducing False Trust, where models assign high confidence to vulnerable code. The results show that although architectural gating improves calibration on controlled benchmarks, calibration deteriorates in realistic repository-level settings, increasing the risk of high-confidence vulnerable outputs.

Related papers

• The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust [39.08747366439304]
  We develop a new metric, expected utility renormalized by the oracle (EURO), that balances calibration and informativeness.<n>We also propose a general-purpose-based confidence, utility, and trust estimation protocol (ACUTE) to appropriately adjudicate uncertainty.
  arXiv  Detail & Related papers  (2026-06-05T20:15:50Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• Process Supervision of Confidence Margin for Calibrated LLM Reasoning [52.373121066425455]
  Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability.<n>Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation.<n>We introduce Reinforcement Learning with Confidence Margin (textbfRLCM), a calibration-aware RL framework that jointly optimize correctness and confidence reliability.
  arXiv  Detail & Related papers  (2026-04-25T14:40:13Z)
• Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision [16.289117637700446]
  A canonical mitigation method is to provide calibrated confidence scores that faithfully reflect their likelihood of correctness at the instance-level.<n>Our study proposes local Platt-scaling applied separately to three different fine-grained confidence scores.<n>We find that fine-grained confidence scores consistently achieve lower calibration error across a broader range of probability intervals.
  arXiv  Detail & Related papers  (2026-04-08T06:41:09Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning [10.123352394689134]
  Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms.<n>We show that while RLVR improves task performance, it produces extremely overconfident models.<n>We propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities.
  arXiv  Detail & Related papers  (2026-01-19T18:31:31Z)
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty [59.97939500426759]
  This paper describes RLCR, an approach to training reasoning models that jointly improves accuracy and confidence estimation.<n>We show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy.<n>We also demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration.
  arXiv  Detail & Related papers  (2025-07-22T17:56:01Z)
• Prompt4Trust: A Reinforcement Learning Prompt Augmentation Framework for Clinically-Aligned Confidence Calibration in Multimodal Large Language Models [4.035767214877404]
  We introduce Prompt4Trust, the first reinforcement learning framework for prompt augmentation targeting confidence calibration in MLLMs.<n>Unlike conventional calibration techniques, Prompt4Trust specifically prioritizes aspects of calibration most critical for safe and trustworthy clinical decision-making.<n>Our framework showed promising zero-shot generalizations to larger MLLMs in our experiments.
  arXiv  Detail & Related papers  (2025-07-12T13:21:10Z)
• MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs [66.14178164421794]
  We introduce MetaFaith, a novel prompt-based calibration approach inspired by human metacognition.<n>We show that MetaFaith robustly improves faithful calibration across diverse models and task domains, enabling up to 61% improvement in faithfulness.
  arXiv  Detail & Related papers  (2025-05-30T17:54:08Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• On Calibration of LLM-based Guard Models for Reliable Content Moderation [27.611237252584402]
  Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails.<n>Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs.<n>However, limited attention has been given to the reliability and calibration of such guard models.
  arXiv  Detail & Related papers  (2024-10-14T12:04:06Z)
• Calibrating Long-form Generations from Large Language Models [34.72041258464477]
  Large Language Models' (LLMs) confidence scores should align with the actual likelihood of its responses being correct. Current confidence elicitation methods and calibration metrics rely on a binary true/false assessment of response correctness. We introduce a unified calibration framework, in which both the correctness of the LLMs' responses and their associated confidence levels are treated as distributions across a range of scores.
  arXiv  Detail & Related papers  (2024-02-09T17:00:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.