Related papers: 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

URL: http://arxiv.org/abs/2607.07980v1
Date: Wed, 08 Jul 2026 23:11:46 GMT
Title: 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse
Abstract summary: Coding agents now author entire pull requests.<n> practitioners sharply disagree about what this does to code review.<n>Agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones.
Score: 15.948323546029714
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built. Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. A motivating observational analysis of public GITHUB activity finds that agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones, yet the direction of these trends flips under different but equally defensible analysis choices, so the traces establish what is changing without explaining why. To recover the mechanisms, we synthesize practitioner discourse at scale into an explanatory theory: we collect 38,709 grey-literature documents (engineering blogs and Reddit threads), filter to those substantively about code review, and code a stratified random sample of 3,100 with an LLM-assisted pipeline, from which we build a causal model of 26 constructs and 67 relationships (64 directed, 3 contested). Its organizing claim is that review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. The theory makes the competing positions explicit and turns "AI is changing code review" into falsifiable propositions with named constructs and moderators. As a secondary contribution, we offer the underlying LLM-assisted, grey-literature theory-building method as a scalable template for software-engineering research, with a public implementation.

Related papers

PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
arXiv Detail & Related papers (2026-09-15T11:08:11Z)
From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality [3.7086626614863984]
We study 1.02 million reviewed pull requests from 207 GitHub projects that transition across three code review eras.<n>Agent-involved collaboration patterns are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption.<n>These findings provide empirical guidance for designing AI-supported code review processes that improve efficiency without weakening review quality.
arXiv Detail & Related papers (2026-07-14T18:48:20Z)
Empirical Study on the Characteristics and Evolution of AI-usage in GitHub Repositories: Evidence from Code Comments [7.213400161126317]
We analyze 35,361 GitHub code comments that explicitly reference AI use and their associated code blocks.<n>We first open-code 500 unique comments and code blocks to derive a taxonomy of AI-assisted development activities.<n>We then annotate the full dataset using two LLM-based classifiers and aggregate predictions with Dawid-Skene expectation-maximization.
arXiv Detail & Related papers (2026-06-05T02:37:44Z)
What Really Improves Mathematical Reasoning: Structured Reasoning Signals Beyond Pure Code [72.9921566968371]
We revisit the claim that code improves reasoning through controlled pretraining experiments on a 10T-token corpus with fine-grained domain separation.<n>Code substantially improves programming ability but does not act as a general reasoning enhancer.<n>Cross-domain structured reasoning traces, such as code-text and math-text mixtures, are better explained.
arXiv Detail & Related papers (2026-05-19T12:37:01Z)
AI for Auto-Research: Roadmap & User Guide [107.0834449839233]
We present an end-to-end analysis of AI across the complete research lifecycle.<n>We identify a sharp, stage-dependent boundary between reliable assistance and unreliable autonomy.<n>We show that greater automation can obscure rather than eliminate failure modes.
arXiv Detail & Related papers (2026-05-18T17:08:26Z)
Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review [2.2807488730605296]
We present a vision for an AI-powered code review workflow combining specialized agents with human-controlled quality.<n>We identify major open challenges for responsible adoption, including reliability, bias, privacy, automation bias, transparency, and evaluation.
arXiv Detail & Related papers (2026-05-17T17:04:21Z)
When AI reviews science: Can we trust the referee? [73.47745294608072]
We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
arXiv Detail & Related papers (2026-04-26T08:03:32Z)
Understanding Dominant Themes in Reviewing Agentic AI-authored Code [6.183483850365225]
We analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories.<n>We find that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight.
arXiv Detail & Related papers (2026-01-27T07:21:09Z)
Early-Stage Prediction of Review Effort in AI-Generated Pull Requests [0.0]
We analyze 33,707 agent-authored PRs from the AIDev dataset across 2,807 repositories.<n>We propose a Circuit Breaker triage model that predicts high-review-effort PRs at creation time.
arXiv Detail & Related papers (2026-01-02T17:18:01Z)
Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
arXiv Detail & Related papers (2025-08-29T08:48:00Z)
Code Review as Decision-Making -- Building a Cognitive Model from the Questions Asked During Code Review [2.8299846354183953]
We build a cognitive model of code review bottom up through thematic, statistical, temporal, and sequential analysis of the transcribed material.<n>The model shows how developers move through two phases during the code review; first an orientation phase to establish context and rationale, then an analytical phase to understand, assess, and plan the rest of the review.
arXiv Detail & Related papers (2025-07-13T14:04:16Z)
Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment [54.62926010621013]
We introduce a novel task, code reasoning, to provide a new perspective for the reasoning abilities of large language models.<n>We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks.<n>We present a new pathway exploration pipeline inspired by human intricate problem-solving methods.
arXiv Detail & Related papers (2025-02-17T10:39:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.