Test Coverage Analysis of Agentic Pull Requests

• URL: http://arxiv.org/abs/2607.18057v1
• Date: Mon, 20 Jul 2026 15:26:30 GMT
• Title: Test Coverage Analysis of Agentic Pull Requests
• Abstract summary: coding agents submit complete pull requests (PRs) with minimal human intervention.<n>As these agents become more prevalent, ensuring the code they generate is adequately tested is critical to preventing regressions.<n>We analyze 4882 agent-generated PRs from the AIDev dataset (532 Java and 4350 Python PRs) produced by five coding agents.
• Score: 13.366199469803922
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI coding agents increasingly submit complete pull requests (PRs) with minimal human intervention, shifting software development from AI-assisted to autonomous workflows. As these agents become more prevalent, ensuring the code they generate is adequately tested, by existing tests or by tests the agents write, is critical to preventing regressions, yet little is known about testing in agentic PRs. To address this gap, we analyze 4882 agent-generated PRs from the AIDev dataset (532 Java and 4350 Python PRs) produced by five coding agents. We study (i) how often agents include test changes and (ii) how well covered are code changes by existing and agent-written tests. Agents include test changes in only 49.6% of PRs that change code under test files. Existing tests provide an incomplete safety net: they cover 61.5% of agents' changed executable lines in Java and only 27.0% in Python, where 64.8% of PRs have no changed line executed by any existing test. Agent-written tests improve coverage over existing tests, but only in a minority of PRs: 35.9% of Java and 22.5% of Python Code + Tests PRs show a coverage gain. Across both languages, error-handling constructs (e.g., try and catch blocks) are the most consistently under-tested, with miss rates reaching 86.0% in Java and 81.0% in Python. These findings motivate coverage-aware development practices, coverage feedback loops for coding agents, and evaluation benchmarks that measure test quality to better help agents reliably test their own code.

Related papers

• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests in Open-Source Projects [2.9713899640333348]
  Integrating AI-powered coding agents into Continuous Integration/Continuous Delivery pipelines has fundamentally altered how software verification is conducted.<n>While these agents successfully automate the test generation, current evaluation benchmarks largely focus on pass-rates rather than the intrinsic quality of the generated tests.<n>We compare 204,673 test artifacts which comprises of 24,941 human-authored files and 179,732 agent-generated files; sourced from the AIDev dataset.
  arXiv  Detail & Related papers  (2026-07-13T18:40:24Z)
• StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns [45.43677974796221]
  We introduce StaminaBench, a benchmark that measures the stamina of coding agents.<n>Unlike the prevailing fraction-tasks metric, this matches real vibecoding where sessions run dozens or hundreds of turns.
  arXiv  Detail & Related papers  (2026-06-17T21:36:09Z)
• Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions [52.50730821321986]
  We present the first large-scale measurement study of version-level risk in large language models (LLMs)<n>We evaluate 10 LLMs on PinTrace, a curated benchmark of 1,000 Stack Overflow programming tasks.<n>Our findings reveal LLM version selection as a first-class, previously overlooked risk surface in LLM-based development.
  arXiv  Detail & Related papers  (2026-05-07T13:52:59Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
  Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
  arXiv  Detail & Related papers  (2026-04-02T01:13:40Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• TDAD: Test-Driven Agentic Development - Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis [0.0]
  TDAD builds a dependency map between source code and tests so that before committing a patch, the agent knows which tests to verify and can self-correct.<n>This paper presents TDAD, an open-source tool that performs pre-change impact analysis for AI coding agents.
  arXiv  Detail & Related papers  (2026-03-18T17:38:22Z)
• Are Coding Agents Generating Over-Mocked Tests? An Empirical Study [2.3625700564650347]
  Coding agents have received significant adoption in software development recently.<n>This paper presents the first study to investigate the presence of mocks in agent-generated tests of real-world software systems.<n>Overall, we find that coding agents are more likely to modify tests and to add mocks to tests than non-coding agents.
  arXiv  Detail & Related papers  (2026-01-30T23:55:25Z)
• Change And Cover: Last-Mile, Pull Request-Based Regression Test Augmentation [20.31612139450269]
  Testing pull requests (PRs) is critical to maintaining software quality.<n>Some PR-modified lines remain untested, leaving a "last-mile" regression test gap.<n>We present ChaCo, an LLM-based test augmentation technique that addresses this gap.
  arXiv  Detail & Related papers  (2026-01-16T02:08:16Z)
• Agentic Property-Based Testing: Finding Bugs Across the Python Ecosystem [34.68658860352019]
  Property-based testing (PBT) is a lightweight formal method, typically implemented as a randomized testing framework.<n>In this work, we demonstrate an LLM-based agent which analyzes Python modules, infers function-specific and cross-function properties from code and documentation, synthesizes and executes PBTs.
  arXiv  Detail & Related papers  (2025-10-10T22:43:54Z)
• RedCode: Risky Code Execution and Generation Benchmark for Code Agents [50.81206098588923]
  RedCode is a benchmark for risky code execution and generation. RedCode-Exec provides challenging prompts that could lead to risky code execution. RedCode-Gen provides 160 prompts with function signatures and docstrings as input to assess whether code agents will follow instructions.
  arXiv  Detail & Related papers  (2024-11-12T13:30:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.