AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection

• URL: http://arxiv.org/abs/2604.11950v1
• Date: Mon, 13 Apr 2026 18:44:02 GMT
• Title: AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection
• Abstract summary: We present AnyPoC, a general multi-agent framework that analyzes and fact-checks a candidate bug report.<n>AnyPoC produces 1.3x more valid PoCs for true-positive bug reports and rejects 9.8x more false-positive bug reports.<n>To date, AnyPoC has discovered 122 new bugs (105 confirmed, 86 already fixed), with 45 generated PoCs adopted as official regression tests.
• Score: 21.99631570872796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent LLM-based agents can identify many candidate bugs in source code, their reports remain static hypotheses that require manual validation, limiting the practicality of automated bug detection. We frame this challenge as a test generation task: given a candidate report, synthesizing an executable proof-of-concept test, or simply a PoC - such as a script, command sequence, or crafted input - to trigger the suspected defect. Automated PoC generation can act as a scalable validation oracle, enabling end-to-end autonomous bug detection by providing concrete execution evidence. However, naive LLM agents are unreliable validators: they are biased toward "success" and may reward-hack by producing plausible but non-functional PoCs or even hallucinated traces. To address this, we present AnyPoC, a general multi-agent framework that (1) analyzes and fact-checks a candidate bug report, (2) iteratively synthesizes and executes a PoC while collecting execution traces, and (3) independently re-executes and scrutinizes the PoC to mitigate hallucination and reward hacking. In addition, AnyPoC also continuously extracts and evolves a PoC knowledge base to handle heterogeneous tasks. AnyPoC operates on candidate bug reports regardless of their source and can be paired with different bug reporters. To demonstrate practicality and generality, we apply AnyPoC, with a simple agentic bug reporter, on 12 critical software systems across diverse languages/domains (many with millions of lines of code) including Firefox, Chromium, LLVM, OpenSSL, SQLite, FFmpeg, and Redis. Compared to the state-of-the-art coding agents, e.g., Claude Code and Codex, AnyPoC produces 1.3x more valid PoCs for true-positive bug reports and rejects 9.8x more false-positive bug reports. To date, AnyPoC has discovered 122 new bugs (105 confirmed, 86 already fixed), with 45 generated PoCs adopted as official regression tests.

Related papers

• CUADebug: Diagnosing and Repairing Computer-Use Agent Failures [65.56066380320107]
  Computer-use agents (CUAs) operate real desktop and web interfaces through screenshots, mouse and keyboard actions, and stateful UI feedback.<n>Unlike text-only agents, CUA failures arise from coupled visual perception, grounding, low-level interaction, task reasoning, and environment dynamics.<n>We introduce CUA Debug, a framework for diagnosing and repairing CUA failures.
  arXiv  Detail & Related papers  (2026-07-31T13:59:45Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Anchored Self-Play for Code Repair [68.0728003844994]
  Code repair is an important capability for language models (LMs)<n>We propose __generator--fixer self-play__, in which a single model is trained with reinforcement learning to generate bugs and fix them.<n>On BugSourceBench, we find that self-play drifts toward difficult but unrealistic bugs.<n>We propose Anchored Self-Play (ASP), which anchors self-play with a small reference set by adding a code-embedding similarity reward for generation and mixing reference bugs into fixer training.
  arXiv  Detail & Related papers  (2026-07-03T17:51:52Z)
• Agentic Model Checking [12.832868209928039]
  We propose a paradigm that couples LLM agents with a bounded model checking backend.<n>We instantiate the approach in BMC-Agent and evaluate it on LLM-generated kernel and compiler code in C and Rust.
  arXiv  Detail & Related papers  (2026-05-20T17:25:52Z)
• Beyond Code Reasoning: Specification-Anchored Auditing of Multi-Implementation Distributed Protocols [1.5229705287183657]
  SPECA is an audit framework that derives explicit, categorized security properties from natural-language specifications and reuses them across implementations.<n>On the RepoAudit benchmark, SPECA reaches 88.9% precision at 100% recall (F1=0.94) and surfaces 12 author-validated bugs beyond ground truth.<n>On the Sherlock Fusaka Audit Contest (10 targets, 366 submissions), SPECA recovers all 15 vulnerabilities expert-augmented and surfaces 4 fix-confirmed bugs.
  arXiv  Detail & Related papers  (2026-04-29T09:57:07Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• SmartPoC: Generating Executable and Validated PoCs for Smart Contract Bug Reports [12.959714248490506]
  SmartPoC is an automated framework that converts textual audit reports into validated test cases.<n>SmartPoC confirms 236 real bugs out of 545 audit findings at a cost of only $0.03 per finding.
  arXiv  Detail & Related papers  (2025-11-17T05:37:20Z)
• ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases [58.411135609139855]
  "Shortcuts" to complete tasks pose significant risks for reliable assessment and deployment of large language models.<n>We introduce ImpossibleBench, a benchmark framework that measures LLM agents' propensity to exploit test cases.<n>As a practical framework, ImpossibleBench is not just an evaluation but a versatile tool.
  arXiv  Detail & Related papers  (2025-10-23T06:58:32Z)
• BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills [59.003563837981886]
  High quality bugs are key to training the next generation of language model based software engineering (SWE) agents.<n>We introduce a novel method for synthetic generation of difficult and diverse bugs.
  arXiv  Detail & Related papers  (2025-10-22T17:58:56Z)
• Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair [7.118712516789191]
  Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry.<n>Showing unlikely patches to developers can lead to substantial noise.<n>We introduce two complementary policies to reduce such noise: bug abstention and patch validation policies.
  arXiv  Detail & Related papers  (2025-10-03T17:53:28Z)
• Eliminating Hallucination-Induced Errors in LLM Code Generation with Functional Clustering [0.0]
  We present functional clustering, a black-box wrapper that eliminates nearly all hallucination-induced errors while providing a tunable confidence score.<n>Our verifier preserves baseline pass@1 on solvable tasks yet slashes the error rate of returned answers from 65% to 2%.<n>Because the method requires only sampling and sandbox execution, it applies unchanged to closed-source APIs and future models.
  arXiv  Detail & Related papers  (2025-05-16T18:19:38Z)
• RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing [8.846583362353169]
  RepoAudit is an autonomous repository-level code auditing agent.<n>It detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%.<n>Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed.
  arXiv  Detail & Related papers  (2025-01-30T05:56:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.