PBT-Bench: Benchmarking AI Agents on Property-Based Testing

• URL: http://arxiv.org/abs/2605.15229v2
• Date: Wed, 20 May 2026 00:07:46 GMT
• Title: PBT-Bench: Benchmarking AI Agents on Property-Based Testing
• Abstract summary: We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
• Score: 29.035258104995204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue. Neither isolates the distinct skill of property-based testing: deriving a semantic invariant from documentation, and then constructing an input-generation strategy precise enough to make a random search reveal the violation. We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries. Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them; the agent must read the library's documentation, identify the relevant invariant, and specify a Hypothesis @given strategy that concentrates mass in the trigger region. Bugs are stratified across three difficulty levels (L1-L3) spanning single-constraint boundary bugs to stateful, cross-function protocol violations. We evaluate eight contemporary LLMs under two prompting regimes (open-ended baseline vs. explicit Hypothesis scaffolding) for three independent runs per configuration. Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%. Hypothesis scaffolding lifts mid-capability models by over 20 percentage points, but yields smaller gains for the strongest models, with two exceptions showing degradation, suggesting the structured prompt can interfere with certain model behaviours rather than complementing them. The hardest bugs prove model-specific: different architectures fail on different problems, leaving persistent gaps that no single model closes. We release the benchmark, harness, and full evaluation corpus to support downstream work on documentation-grounded semantic reasoning.

Related papers

• Evaluating Agentic Code Repair Capabilities in Distributed Systems [12.535741000126803]
  We introduce DDBench, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems.<n>The evaluation of ten LLMs on DDBench reveals several findings.
  arXiv  Detail & Related papers  (2026-08-14T19:59:56Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects [2.9713899640333348]
  Integrating AI-powered coding agents into Continuous Integration/Continuous Delivery pipelines has fundamentally altered how software verification is conducted.<n>While these agents successfully automate the test generation, current evaluation benchmarks largely focus on pass-rates rather than the intrinsic quality of the generated tests.<n>We compare 204,673 test artifacts which comprises of 24,941 human-authored files and 179,732 agent-generated files; sourced from the AIDev dataset.
  arXiv  Detail & Related papers  (2026-07-13T18:40:24Z)
• DeFAb: A Verifiable Benchmark for Defeasible Abduction in Foundation Models [6.628401122676601]
  A rule-based logic solver resolves every instance in our benchmark in under 50 microseconds with 100% accuracy.<n>We introduce DeFAb (Defeasible Abduction Benchmark), a dataset and generation pipeline.
  arXiv  Detail & Related papers  (2026-06-17T00:13:40Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills [59.003563837981886]
  High quality bugs are key to training the next generation of language model based software engineering (SWE) agents.<n>We introduce a novel method for synthetic generation of difficult and diverse bugs.
  arXiv  Detail & Related papers  (2025-10-22T17:58:56Z)
• Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem [34.68658860352019]
  Property-based testing (PBT) is a lightweight formal method, typically implemented as a randomized testing framework.<n>In this work, we demonstrate an LLM-based agent which analyzes Python modules, infers function-specific and cross-function properties from code and documentation, synthesizes and executes PBTs.
  arXiv  Detail & Related papers  (2025-10-10T22:43:54Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Data Leakage and Redundancy in the LIT-PCBA Benchmark [0.0]
  LIT-PCBA is widely used to benchmark virtual screening models.<n>We find extensive data leakage and molecular redundancy across its models.<n>Almost all published results on LIT-PCBA are undermined.
  arXiv  Detail & Related papers  (2025-07-29T00:23:45Z)
• CITADEL: Context Similarity Based Deep Learning Framework Bug Finding [37.985497279785235]
  Existing DL framework testing tools are inefficient, generating hundreds of test cases with few trigger bugs.<n>We propose Citadel, a method that accelerates bug finding in terms of efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-18T01:51:16Z)
• SBEST: Spectrum-Based Fault Localization Without Fault-Triggering Tests [17.90798133817018]
  This study investigates the feasibility of using stack traces from crash reports as proxies for fault-triggering tests in Spectrum-Based Fault localization.<n>We propose SBEST, a novel approach that integrates stack trace information with test coverage data to perform fault localization when fault-triggering tests are missing.
  arXiv  Detail & Related papers  (2024-05-01T15:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.