What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

• URL: http://arxiv.org/abs/2609.12708v1
• Date: Fri, 11 Sep 2026 11:02:42 GMT
• Title: What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code
• Abstract summary: We compare human-written and AI-generated code at scale.<n>We characterize structural complexity and statistical naturalness.<n>We release CQBench, a benchmark of 27,346 issueprone tasks.
• Score: 4.478789600295493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whether their code differs from human code in the quality dimensions dominating lifecycle cost. We compare human-written and AI-generated code at scale: 787,562 function pairs across Python, Java, and C, each human function mined from open-source repositories paired with implementations generated from its docstring by three AI assistants (OpenAI GPT models, DeepSeek-Coder, Qwen2.5-Coder). We characterize structural complexity and statistical naturalness, and map static-analysis findings onto Orthogonal Defect Classification for defects and the Common Weakness Enumeration for vulnerabilities, making authors and languages directly comparable. AI-generated code is structurally compressed and stylistically templated: roughly half the size and branching of human code, clustering apart at the style level. Defect profiles differ in kind: human code concentrates issues of mature codebases, AI code repetitive boilerplate; security is language-dependent, with LLMs producing more, and more severe, findings in Python and Java but fewer high-severity memory-safety findings than humans in C. Once size is controlled for, complexity metrics carry little signal, while naturalness separates authors. Finally, we release CQBench, a benchmark of 27,346 issue-prone tasks with baselines and an evaluation pipeline for quality assurance and security testing.

Related papers

• The Illusion of Safety: Multi-Tier Verification of AI vs. Human C++ Code [10.544313262813812]
  Large language models increasingly generate C++, a memory-unsafe language where a single overlooked violation can become an exploitable bug.<n>Most security evaluations of AI-generated code rely on static analysis alone, which flags warnings without confirming runtime violations or reasoning about untested paths.<n>We ask whether AI-generated C++ is measurably less safe than human-written code, and whether common verification tools agree on the risk.
  arXiv  Detail & Related papers  (2026-06-30T19:48:35Z)
• HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection [1.6385993554828697]
  HybridCodeAuthorship is a novel benchmark of Python code files with interleaved human- and AI-authored lines of code.<n>We then benchmark the performance of two state-of-the-art AI-generated code detection algorithms at both the line- and chunk-level.
  arXiv  Detail & Related papers  (2026-06-10T19:21:19Z)
• FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale [73.57353294235223]
  We introduce FrontierSmith, an automated system for evolving open-ended problems from existing closed-ended coding tasks.<n>On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models.<n>The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones.
  arXiv  Detail & Related papers  (2026-05-14T06:39:42Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• Code for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics [6.108440460022983]
  We investigate the concept of AI-friendly code' via a dataset of 5,000 Python files from competitive programming.<n>Our findings confirm that human-friendly code is also more compatible with AI tooling.<n>These results suggest that organizations can use CodeHealth to guide where AI interventions are lower risk and where additional human oversight is warranted.
  arXiv  Detail & Related papers  (2026-01-05T15:23:55Z)
• Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexity [4.478789600295493]
  We present a large-scale comparison of code authored by human developers and three state-of-the-art LLMs, i.e., ChatGPT, DeepSeek-Coder, and Qwen-Coder.<n>Our evaluation spans over 500k code samples in two widely used languages, Python and Java, classifying defects via Orthogonal Defect Classification and security vulnerabilities using the Common Weaknession.<n>We find that AI-generated code is generally simpler and more prone to unused constructs and hardcoded, while human-written code exhibits greater structural complexity and a higher concentration of maintainability issues.
  arXiv  Detail & Related papers  (2025-08-29T13:51:28Z)
• MultiAIGCD: A Comprehensive dataset for AI Generated Code Detection Covering Multiple Languages, Models,Prompts, and Scenarios [0.0]
  We introduce MultiAIGCD, a dataset for AI-generated code detection for Python, Java, and Go.<n>Overall, MultiAIGCD consists of 121,271 AI-generated and 32,148 human-written code snippets.
  arXiv  Detail & Related papers  (2025-07-29T11:16:55Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding [49.56049319037421]
  KodCode is a synthetic dataset that addresses the persistent challenge of acquiring high-quality, verifiable training data.<n>It comprises question-solution-test triplets that are systematically validated via a self-verification procedure.<n>This pipeline yields a large-scale, robust and diverse coding dataset.
  arXiv  Detail & Related papers  (2025-03-04T19:17:36Z)
• Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting [78.48355455324688]
  We propose a novel zero-shot synthetic code detector based on the similarity between the original code and its LLM-rewritten variants.<n>Our results demonstrate a significant improvement over existing SOTA synthetic content detectors.
  arXiv  Detail & Related papers  (2024-05-25T08:57:28Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.