Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs

• URL: http://arxiv.org/abs/2606.20243v2
• Date: Mon, 22 Jun 2026 11:00:39 GMT
• Title: Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs
• Abstract summary: We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation.<n>Phoenix decomposes the work across six specialized agents.<n>A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation, combining seven layered safety controls with a baseline-aware test evaluation strategy. Phoenix decomposes the work across six specialized agents. Planner, reproducer, coder, tester, failure analyst and Pull Request (PR) agent, all coordinated by a label-based GitHub webhook state machine. Every change is checked against a baseline test run before a pull request is opened. On a 24-instance slice of SWE-bench Lite. run on the production webhook path, Phoenix oracle-resolves 75% of instances with no pass-to-pass regressions on successful runs; this curated slice is not directly comparable to full-split leaderboard results, and we discuss the limits of the comparison. A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation (CP; mean 122s on the hard tier). Manual inspection shows that about half of the resulting pull requests are well-targeted fixes. The other half place code at incorrect paths, a planner localization limitation we are addressing with retrieval. We also report the deployment failure modes (WAF filtering, token expiry, permission boundaries, flaky CI) that motivated each safety mechanism.

Related papers

• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents [63.65682461568621]
  We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness.<n>SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints.<n>Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance.
  arXiv  Detail & Related papers  (2026-09-03T17:53:34Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns [45.43677974796221]
  We introduce StaminaBench, a benchmark that measures the stamina of coding agents.<n>Unlike the prevailing fraction-tasks metric, this matches real vibecoding where sessions run dozens or hundreds of turns.
  arXiv  Detail & Related papers  (2026-06-17T21:36:09Z)
• Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench [33.69401287706814]
  We ask whether agentic AI systems built for software engineering transfer to realistic hardware engineering.<n>We introduce textbfPhoenix-bench, a synchronized corpus of 511 verified Verilator instances from 114 GitHub repositories.<n>Using Phoenix-bench we run a uniform evaluation of four commercial agents and eight open-source agentic structures.
  arXiv  Detail & Related papers  (2026-05-13T14:14:54Z)
• WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation [88.10947115397971]
  This work presents WildClawBench, a native-runtime benchmark of 60 human-authored, bilingual, multimodal tasks spanning six thematic categories.<n>Each task averages roughly 8 minutes of wall-clock time and over 20 tool calls.<n> Grading is hybrid, combining deterministic rule-based checks, environment-state auditing of side effects, and an LLM/VLM judge for semantic verification.
  arXiv  Detail & Related papers  (2026-05-11T17:49:43Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering [3.2126925586839623]
  We introduce execution-grounded verification as a first-class principle.<n>We instantiate this principle in AGENTFORGE, a multi-agent framework.<n>AGENTFORGEtoken achieves 40.0% resolution on SWE-BENCH Lite.
  arXiv  Detail & Related papers  (2026-04-13T13:51:13Z)
• DRS-OSS: LLM-Driven Diff Risk Scoring Tool for PR Risk Prediction [3.17083738531489]
  Diff Risk Scoring (DRS) estimates the likelihood that a diff will introduce a defect, enabling better review prioritization, test planning, and CI/CD gating.<n>DRS-OSS is an open-source DRS system equipped with a public API, web UI, and GitHub plugin.
  arXiv  Detail & Related papers  (2025-11-26T22:52:18Z)
• InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution [31.874379525390967]
  InfCode is an adversarial multi-agent framework for automated repository-level issue resolution.<n>InfCode iteratively refines both tests and patches through adversarial interaction between a Test Patch Generator and a Code Patch Generator.<n> Experiments on SWE-bench Lite and SWE-bench Verified using models such as DeepSeek-V3 and Claude 4.5 Sonnet show that InfCode consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-11-20T03:04:21Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution [47.850418420195304]
  Large Language Models (LLMs) have shown promise in code generation but face difficulties in resolving GitHub issues. We propose a novel Multi-Agent framework for GitHub Issue reSolution, MAGIS, consisting of four agents customized for software evolution.
  arXiv  Detail & Related papers  (2024-03-26T17:57:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.