Trust-Calibrated Code Review: A Participatory Design Study of Review Workflows for LLM-Generated Multi-File Changes

• URL: http://arxiv.org/abs/2606.01969v1
• Date: Mon, 01 Jun 2026 09:32:25 GMT
• Title: Trust-Calibrated Code Review: A Participatory Design Study of Review Workflows for LLM-Generated Multi-File Changes
• Abstract summary: Reviewing multi-file changes generated by LLM-based agents is a trust-calibration problem rather than a diffing problem.<n>The study yielded a three-level review workflow supported by seven design constructs.<n>These findings suggest that the design constructs indicate a positive direction for future tool development.
• Score: 1.8912594675203016
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Developers increasingly review multi-file code changes generated by LLM-based agents, yet no validated end-to-end workflow or IDE tooling design exists for this scenario. Aims: We investigate (RQ1) the challenges developers face when reviewing LLM-generated multi-file changes and (RQ2) how developers envision effective workflows for this task. Method: In collaboration with JetBrains, we conducted a participatory design study structured using the double-diamond design process with Discover, Define, Develop, and Deliver phases. Industry practitioners participated in the Discover phase (N=17); seven of these returned for the Develop phase. The Define phase was an author-led synthesis. The Deliver phase produced a conceptual design and a high-fidelity semi-interactive prototype evaluated through a follow-up survey with N=43 practitioners. Results: Participants identified trust-calibration as the central challenge. The study yielded a three-level review workflow (overview, file-analysis, code snippet review) supported by seven design constructs (chunk, risk-per-line, risk-per-file, judge, walk-through, zooming in/out, and security cage). In the validation survey, all three workflow levels scored above the neutral midpoint (means 3.50--3.91 on a five-point scale). Of the respondents, 63% expected reduced overall review effort, and 52% reduced trust-assessment effort, relative to their current tools. These findings suggest that the design constructs indicate a positive direction for future tool development. Conclusions: Reviewing LLM-generated multi-file changes is a trust-calibration problem rather than a diffing problem. The three-level workflow and the seven constructs we report give tool designers a conceptual framework for building AI-ready code review tools that surface risk and confidence signals at the granularity at which developers allocate attention.

Related papers

• MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding [83.0805548642958]
  MTAC-IFBench is a comprehensive benchmark for instruction-following in software development.<n>It features multi-turn progressive software development instructions with diverse constraints spanning 6 primary and 18 secondary categories.<n>MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with their performance degrading rapidly as the interaction session grows longer.
  arXiv  Detail & Related papers  (2026-09-14T03:57:37Z)
• Beyond the YAML File: Understanding Real-World GitHub Actions Workflow Adoption [3.1994648513393873]
  GitHub Actions (GHA) are emerging as a dominant automation platform.<n>We quantitatively analyze 258,300 workflow run records from 952 repositories to understand how maintainers and contributors interact with workflow results.<n>We identify three distinct failure response patterns, observe that higher usage of GHA correlates with lower failure rates, and uncover a configuration-usage gap.
  arXiv  Detail & Related papers  (2026-04-19T23:23:59Z)
• From Helpful to Trustworthy: LLM Agents for Pair Programming [0.0]
  LLM-based coding agents are increasingly used to generate code, tests, and documentation.<n>Still, their outputs can be plausible yet misaligned with developer intent and provide limited evidence for review in evolving projects.<n>This research proposes a systematic study of multi-agent LLM pair programming that externalizes intent and uses development tools for iterative validation.
  arXiv  Detail & Related papers  (2026-04-11T17:39:57Z)
• ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development [49.63491095660809]
  ProjDevBench is an end-to-end benchmark that provides project requirements to coding agents and evaluates the resulting repositories.<n>We curate 20 programming problems across 8 categories, covering both concept-oriented tasks and real-world application scenarios.<n>Our evaluation reports an overall acceptance rate of 27.38%: agents handle basic functionality but struggle with complex system design, time optimization, and resource management.
  arXiv  Detail & Related papers  (2026-02-02T05:17:23Z)
• Understanding Specification-Driven Code Generation with LLMs: An Empirical Study Design [2.687678248171195]
  Large Language Models (LLMs) are increasingly integrated into software development, yet their behavior in structured, specification-driven processes remains poorly understood.<n>This paper presents an empirical study design using CURRANTE, a Visual Studio Code extension that enables a human-in-the-loop workflow for LLM-assisted code generation.<n>The study aims to analyze how human intervention in specification and test refinement influences the quality and dynamics of LLM-generated code.
  arXiv  Detail & Related papers  (2026-01-07T12:46:57Z)
• Beyond Greenfield: The D3 Framework for AI-Driven Productivity in Brownfield Engineering [0.0]
  Brownfield engineering work involving legacy systems, incomplete documentation, and fragmented architectural knowledge poses unique challenges for the effective use of large language models (LLMs)<n>This paper introduces the Discover-Define-Deliver (D3) Framework, a disciplined LLM-assisted workflow that combines role-separated prompting strategies with applied best practices for navigating ambiguity in brownfield systems.<n> Respondents reported perceived improvements in clarity, quality, documentation and cognitive load, along with self-estimated productivity gains.
  arXiv  Detail & Related papers  (2025-12-01T00:26:41Z)
• Evaluating Software Process Models for Multi-Agent Class-Level Code Generation [5.545076518491288]
  Large Language Models (LLMs) are increasingly used to automate software development.<n>This work examines how process structure and role shape multi-agent specialization for class-level code generation.
  arXiv  Detail & Related papers  (2025-11-12T22:53:12Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• FlowBench: Revisiting and Benchmarking Workflow-Guided Planning for LLM-based Agents [64.1759086221016]
  We present FlowBench, the first benchmark for workflow-guided planning. FlowBench covers 51 different scenarios from 6 domains, with knowledge presented in diverse formats. Results indicate that current LLM agents need considerable improvements for satisfactory planning.
  arXiv  Detail & Related papers  (2024-06-21T06:13:00Z)
• SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents [50.82665351100067]
  FlowGen is a code generation framework that emulates software process models based on multiple Large Language Model (LLM) agents. We evaluate FlowGenScrum on four benchmarks: HumanEval, HumanEval-ET, MBPP, and MBPP-ET.
  arXiv  Detail & Related papers  (2024-03-23T14:04:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.