The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code

• URL: http://arxiv.org/abs/2607.00107v2
• Date: Sun, 05 Jul 2026 21:34:36 GMT
• Title: The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code
• Abstract summary: Large language models increasingly generate C++, a memory-unsafe language where a single overlooked violation can become an exploitable bug.<n>Most security evaluations of AI-generated code rely on static analysis alone, which flags warnings without confirming runtime violations or reasoning about untested paths.<n>We ask whether AI-generated C++ is measurably less safe than human-written code, and whether common verification tools agree on the risk.
• Score: 10.544313262813812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models increasingly generate C++, a memory-unsafe language where a single overlooked violation can become an exploitable bug. Yet most security evaluations of AI-generated code rely on static analysis alone, which flags warnings without confirming runtime violations or reasoning about untested paths. We ask whether AI-generated C++ is measurably less safe than human-written code, and whether common verification tools agree on the risk. We introduce VULBENCH-CPP, a benchmark of 8,918 C++ programs from three open-weight LLMs (Gemma 3 27B IT, LLaMA 3.3 70B Instruct, Qwen 2.5 Coder 32B Instruct) and human authors across 851 competitive-programming tasks. Each program is annotated by four verification tiers: functional testing, static analysis (cppcheck, clang-tidy), dynamic analysis (ASan/UBSan), and bounded model checking (ESBMC). Accounting for the correlation among solutions to a shared task, we find that AI-generated code is roughly twice as likely as human code to trigger a confirmed runtime violation, even after controlling for code length and test pass-rate. Under static analysis the two look equally safe, but this is misleading: the apparent similarity reflects code length rather than real safety, and the tiers detect largely different classes of violation, so no single tier is sufficient. The gap is consistent across independent generations.

Related papers

• What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code [4.478789600295493]
  We compare human-written and AI-generated code at scale.<n>We characterize structural complexity and statistical naturalness.<n>We release CQBench, a benchmark of 27,346 issueprone tasks.
  arXiv  Detail & Related papers  (2026-09-11T11:02:42Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• AICD Bench: A Challenging Benchmark for AI-Generated Code Detection [91.21422299346199]
  AICD Bench is the most comprehensive benchmark for AI-generated code detection.<n>It spans $emph2M examples$, $emph77 models$ across $emph11 families$, and $emph9 programming languages$, including recent reasoning models.
  arXiv  Detail & Related papers  (2026-02-02T13:24:14Z)
• Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity [4.478789600295493]
  We present a large-scale comparison of code authored by human developers and three state-of-the-art LLMs, i.e., ChatGPT, DeepSeek-Coder, and Qwen-Coder.<n>Our evaluation spans over 500k code samples in two widely used languages, Python and Java, classifying defects via Orthogonal Defect Classification and security vulnerabilities using the Common Weaknession.<n>We find that AI-generated code is generally simpler and more prone to unused constructs and hardcoded, while human-written code exhibits greater structural complexity and a higher concentration of maintainability issues.
  arXiv  Detail & Related papers  (2025-08-29T13:51:28Z)
• Comparative Analysis of the Code Generated by Popular Large Language Models (LLMs) for MISRA C++ Compliance [0.0]
  The software development for safety-critical systems requires rigorous engineering practices and adherence to certification standards like DO-178C for avionics.<n> DO-178C is a guidance document which requires compliance to well-defined software coding standards like MISRA C++.<n>I have conducted a comparative analysis of the C++ code generated by popular LLMs for compliance with MISRA C++.
  arXiv  Detail & Related papers  (2025-06-30T05:53:45Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• A Comparative Study of Fuzzers and Static Analysis Tools for Finding Memory Unsafety in C and C++ [24.60320701097142]
  We present an empirical analysis of five static analyzers and 13 fuzzers, applied to over 100 known security vulnerabilities in C/C++ programs.<n>We find that both techniques discover different types of bugs, but there are clear winners for each.
  arXiv  Detail & Related papers  (2025-05-28T07:22:29Z)
• Towards Exception Safety Code Generation with Intermediate Representation Agents Framework [54.03528377384397]
  Large Language Models (LLMs) often struggle with robust exception handling in generated code, leading to fragile programs that are prone to runtime errors.<n>We propose Seeker, a novel multi-agent framework that enforces exception safety in LLM generated code through an Intermediate Representation (IR) approach.<n>Seeker decomposes exception handling into five specialized agents: Scanner, Detector, Predator, Ranker, and Handler.
  arXiv  Detail & Related papers  (2024-10-09T14:45:45Z)
• GWP-ASan: Sampling-Based Detection of Memory-Safety Bugs in Production [30.534320345970286]
  heap-use-after-free and heap-buffer-overflow bugs remain the primary problem for security, reliability, and developer productivity for applications written in C or C++. This paper describes a family of tools that detect these two classes of memory-safety bugs, while running in production, at near-zero overhead.
  arXiv  Detail & Related papers  (2023-11-15T21:41:53Z)
• A Static Evaluation of Code Completion by Large Language Models [65.18008807383816]
  Execution-based benchmarks have been proposed to evaluate functional correctness of model-generated code on simple programming problems. static analysis tools such as linters, which can detect errors without running the program, haven't been well explored for evaluating code generation models. We propose a static evaluation framework to quantify static errors in Python code completions, by leveraging Abstract Syntax Trees.
  arXiv  Detail & Related papers  (2023-06-05T19:23:34Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.