Open-Source Intelligence for Code Provenance and the Security Patterns that Separate Human and Large-Language-Model Implementations of Common Programming Tasks

• URL: http://arxiv.org/abs/2607.12524v1
• Date: Tue, 14 Jul 2026 08:59:42 GMT
• Title: Open-Source Intelligence for Code Provenance and the Security Patterns that Separate Human and Large-Language-Model Implementations of Common Programming Tasks
• Abstract summary: Developers now draw code from two very different sources.<n>Can the provenance of a code snippet be recovered from the code itself?<n>Do the two sources differ in the security patterns they adopt for the same task?
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Developers now draw code from two very different sources, the accumulated human answers on sites such as Stack Overflow and the output of large language models. We ask two questions about that split. First, can the provenance of a code snippet be recovered from the code itself, and second, do the two sources differ in the security patterns they adopt for the same task. Using only open sources, a public gateway of open-weight language models and the public Stack Overflow API, we build a fully reproducible pipeline that collects real implementations of 31 security-sensitive programming tasks, among them OAuth with PKCE, JWT verification, password hashing, and SQL access, from 9 language models and from human answers, and scores every sample with deterministic security and style detectors. On 528 real samples we train a cross-validated classifier that recovers human versus model provenance with 93 percent accuracy against a 78 percent baseline, and a 7-way classifier that attributes a sample to the specific model that wrote it at 48 percent. We then report where the sources diverge on security, which patterns models adopt more often than the human corpus and which they inherit from it. Running the same tasks in Python, JavaScript, Go, and Java, we find the security divergence holds in every language while the provenance boundary is partly language-specific and does not transfer symmetrically between them. A vulnerability repair case study, in which models are handed insecure code and asked to fix it, finds a 77 percent repair rate across 21 seeds and 12 weakness classes, but a recurring partial-fix failure in which the model removes the insecure pattern without adding the correct defense. The pipeline is data driven, so any new task or language is added as a single specification entry, and a fail-closed checker re-derives every number in this paper from the stored data.

Related papers

• SPARK: Security Knowledge Priming and Representation-Guided Knowledge Activation for LLM-based Secure Code Generation [18.3981349747131]
  Prior literature attributes this limitation to a lack of security expertise.<n>We argue that pretraining corpora are already rich in security material.<n>We present SPARK, an inference-time security harness that activates this latent knowledge without any retraining.
  arXiv  Detail & Related papers  (2026-06-15T05:43:37Z)
• Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs [0.0]
  Best model reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline.<n>Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage.<n>Gemma-2-27B-IT achieves the highest all-language lint pass rate.
  arXiv  Detail & Related papers  (2026-06-07T21:10:30Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• GPT, But Backwards: Exactly Inverting Language Model Outputs [7.736516533045638]
  We present the Sparse One-hot Discrete Adam (SODA) algorithm, a search-based inversion method that can accurately reconstruct the input text.<n>Experiments demonstrate for the first time that exact language model inversion is possible on both natural language and random inputs.
  arXiv  Detail & Related papers  (2025-07-02T13:20:30Z)
• Decompiling Smart Contracts with a Large Language Model [51.49197239479266]
  Despite Etherscan's 78,047,845 smart contracts deployed on (as of May 26, 2025), a mere 767,520 ( 1%) are open source.<n>This opacity necessitates the automated semantic analysis of on-chain smart contract bytecode.<n>We introduce a pioneering decompilation pipeline that transforms bytecode into human-readable and semantically faithful Solidity code.
  arXiv  Detail & Related papers  (2025-06-24T13:42:59Z)
• Traces of Memorisation in Large Language Models for Code [16.125924759649106]
  Large language models for code are commonly trained on large unsanitised corpora of source code scraped from the internet. We compare the rate of memorisation with large language models trained on natural language. We find that large language models for code are vulnerable to data extraction attacks, like their natural language counterparts.
  arXiv  Detail & Related papers  (2023-12-18T19:12:58Z)
• Paloma: A Benchmark for Evaluating Language Model Fit [112.481957296585]
  Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.<n>We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains.
  arXiv  Detail & Related papers  (2023-12-16T19:12:45Z)
• Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense [56.077252790310176]
  We present a paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking. We introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider.
  arXiv  Detail & Related papers  (2023-03-23T16:29:27Z)
• Software Vulnerability Prediction Knowledge Transferring Between Programming Languages [2.3035725779568583]
  We propose a transfer learning technique to leverage available datasets and generate a model to detect common vulnerabilities in different programming languages. We use C source code samples to train a Convolutional Neural Network (CNN) model, then, we use Java source code samples to adopt and evaluate the learned model. The results show that proposed model detects vulnerabilities in both C and Java codes with average recall of 72%.
  arXiv  Detail & Related papers  (2023-03-10T19:21:52Z)
• CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models [58.27254444280376]
  Large language models (LLMs) for automatic code generation have achieved breakthroughs in several programming tasks. Training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the language models to learn these vulnerabilities and propagate them during the code generation procedure.
  arXiv  Detail & Related papers  (2023-02-08T11:54:07Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.