From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts

• URL: http://arxiv.org/abs/2608.09567v1
• Date: Mon, 10 Aug 2026 13:05:02 GMT
• Title: From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts
• Abstract summary: Security research artifacts are increasingly produced by agent-driven vulnerabilities.<n>The gap between emphpublicly available, emphrunnable, emphsignal-producing, and emphsemantically confirmed artifacts is poorly measured.<n>We conduct a pre-registered consensus audit of this literature.
• Score: 4.093009863847175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Security research artifacts---repositories, PoC exploits, and validation pipelines---are increasingly produced by LLM/agent-driven vulnerability workflows, yet the gap between \emph{publicly available}, \emph{runnable}, \emph{signal-producing}, and \emph{semantically confirmed} artifacts is poorly measured. We conduct a pre-registered reproducibility audit of this literature. A search covering 2023--2026 with dual screening yields a 104-paper consensus corpus, of which 59 papers (56.7\%) have a publicly reachable artifact. We execute an 18-paper sample at R0/R1 and all 102 cases of the anchor benchmark (arXiv:2509.24037), with patched-counterfactual verdicts on 30 signal-producing cases and matched-negative-control verdicts on 19. Three findings stand out. First, 58/102 (56.9\%) anchor cases contain a script-internal CVE identifier that diverges from the declared directory CVE. Second, only 10/18 (55.6\%) paper-level artifacts complete their declared workflow at R0, rising to 11/18 (61.1\%) after environment-only R1 repair. Third, artifact-embedded oracles prove unreliable: 20/30 patched-counterfactual audits still produce the claimed signal on the patched build, 7/19 matched negative controls still trigger on benign input, and the oracle confusion matrix has sensitivity 60\% and specificity 45\%. A trigger on the vulnerable build is not evidence of CVE-specific reproduction without a clean patched counterfactual. These are exploratory results from a pre-registered protocol, and our protocol---pre-registered post-conditions, R0/R1 repair ladder, G1--G3 semantic evidence levels, and patched-counterfactual oracles---is a reusable template for the security reproducibility community.

Related papers

• IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications [52.570108663867046]
  A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation.<n>We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions.<n>We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references.
  arXiv  Detail & Related papers  (2026-09-09T17:59:04Z)
• EduPluginBench: Executable Assurance for AI-Generated Educational Plugins [0.0]
  We introduce negative provenance, an executable benchmark and staged admission method for generated plugins in software ecosystems.<n>A frozen transfer study of 600 unmodified generations from two current coding models found that 300/600 parsed, but none passed P0 or achieved P0-P4 conformance.<n>An earlier 540-generation diagnostic found that post-hoc bounded repair yielded 112 P0 passes, all nonconforming, with recall increasing from 13.4% to 100%.
  arXiv  Detail & Related papers  (2026-08-01T16:22:35Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Evidence-Grounded Verified Agentic Reasoning: A Path Toward Eliminating LLM Hallucination in Empirical Inference via Tool-Attested Kernel Proofs [0.564046562677526]
  We show how a Lean 4-based tool-calling architecture can be used to govern empirical claims.<n>Every verified output structurally descends from an attested tool call and a kernel-checked chain of valid inference.<n>A formal sidecar makes the target proposition, source scope, evidence boundary, proof obligation, and abstention condition auditable.<n>Over time, typed sidecars in datasets, APIs, public records, and AI-generated documents can amortize this formalization burden into reusable infrastructure.
  arXiv  Detail & Related papers  (2026-07-14T11:33:44Z)
• Conductance-Repair Evidence Graphs for Prospective Security Retrieval [0.2864713389096699]
  We introduce conductance-repair evidence graphs, a timestamped framework in which retrieval is performed over a temporal admissibility mask and missing channels are widened by a deterministic graph-flow recurrence rather than by a predictor.<n>The current artifact materializes 30 deduplicated public security records, 57 terms, and 58 withheld admissible document-term edges.<n>Under random edge withholding, conductance repair changes recall@(k) from 0.017 to 0.069 and average precision from 0.062 to 0.060, while a synthetic security fixture improves recall@(k) from 0.055 to 0.099.
  arXiv  Detail & Related papers  (2026-07-05T01:03:15Z)
• ContractBench: Can LLM Agents Preserve Observation Contracts? [9.057486468322933]
  We show that observation contract compliance is an emergent, regression-prone capability.<n>We introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two failure modes.<n>We report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%; (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B; (iii) non-monotonic scaling across the GPT-5 family: agentic post-training can erode
  arXiv  Detail & Related papers  (2026-05-17T06:37:04Z)
• How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
  The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
  arXiv  Detail & Related papers  (2026-05-08T09:30:31Z)
• Guaranteeing Knowledge Integration with Joint Decoding for Retrieval-Augmented Generation [45.13215113490545]
  GuarantRAG is a framework that explicitly decouples reasoning from evidence integration.<n> Experiments on five QA benchmarks demonstrate that GuarantRAG improves accuracy by up to 12.1%.
  arXiv  Detail & Related papers  (2026-04-09T09:52:21Z)
• Credential Leakage in LLM Agent Skills: A Large-Scale Empirical Study [51.717224133855886]
  Third-party skills extend LLM agents with powerful capabilities but often handle sensitive credentials in privileged environments.<n>We analyze 17,022 skills (sampled from 170,226 on SkillsMP) using static analysis, sandbox testing, and manual inspection.<n>We identify 520 vulnerable skills with 1,708 issues and derive a taxonomy of 10 leakage patterns (4 accidental and 6 adversarial)
  arXiv  Detail & Related papers  (2026-04-03T14:50:16Z)
• What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs [46.325755802511026]
  We developLM, a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model.<n>LM successfully identified 111 of 5,140 recent Linux kernel patches addressing OOB or UAF vulnerabilities, with 90 true positives confirmed by manual verification.
  arXiv  Detail & Related papers  (2025-09-26T18:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.