Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench

• URL: http://arxiv.org/abs/2605.15226v1
• Date: Wed, 13 May 2026 14:14:54 GMT
• Title: Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
• Abstract summary: We ask whether agentic AI systems built for software engineering transfer to realistic hardware engineering.<n>We introduce textbfPhoenix-bench, a synchronized corpus of 511 verified Verilator instances from 114 GitHub repositories.<n>Using Phoenix-bench we run a uniform evaluation of four commercial agents and eight open-source agentic structures.
• Score: 33.69401287706814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We ask whether agentic AI systems built for software engineering transfer to realistic hardware engineering. Existing hardware LLM benchmarks isolate sub-tasks but none jointly requires repository navigation, hierarchy-aware localization, Electronic Design Automation (EDA) executable verification, and maintenance-style patching. We introduce \textbf{Phoenix-bench}, a synchronized corpus of 511 verified Verilator instances from 114 GitHub repositories, each shipped with the developer patch, design-flow labels, fail-to-pass and pass-to-pass testbenches, and a Docker-pinned EDA environment so resolved-rate differences reflect agent behavior rather than toolchain availability. Using Phoenix-bench we run a uniform evaluation of four commercial agents and eight open-source agentic structures across four LLM backbones, plus two diagnostic interventions (file-level oracle localization and one round of testbench-log feedback). Three findings emerge. (i)~Software and hardware are fundamentally different engineering tasks: the same agent loses 37\% to 58\% from SWE-bench Verified to Phoenix-bench because hardware bugs propagate across parallel instantiated modules through signal flow rather than along a software-style call graph, and software-tuned agents stop at the symptom file instead of tracing back through the instantiation chain. (ii)~Failures concentrate on design control-flow / finite state machine (FSM) bugs, verification testbench bugs, and hard cases that demand cross-hierarchy signal-flow tracking and coordinated multi-file edits. (iii)~Localization granularity matters far more than localization itself: a perfect file-level oracle yields only $+1.4$\% because the agent then breaks files that did not need editing, while a single round of test case feedback lifts resolved rate by $42$\% to $45$\% because the test case tells \emph{where} the bug is and \emph{what} the fix has to look like.

Related papers

• Taming the Drift: Context-aware Repair of Dockerfile Drift during Software Evolution [52.50730821321986]
  Cadre is a context-aware framework for automated Dockerfile drift repair.<n>CDG maps each Dockerfile instruction to its file-level and inter-instruction dependencies.<n>DodeX mines real-world Dockerfile drift instances from GitHub Actions CI logs.
  arXiv  Detail & Related papers  (2026-07-14T09:12:26Z)
• Rise From The Ashes: LLM-based Static Analysis for Deep Learning Framework Bugs [29.26041624588595]
  Deep learning (DL) frameworks are critical AI infrastructures that often hide bugs with serious security implications.<n> Static analysis is a natural complement because it can detect bugs without execution, offering fast and scalable testing.<n>We present Phoenix, the first LLM-based static analysis technique for DL frameworks.
  arXiv  Detail & Related papers  (2026-07-01T07:44:25Z)
• Phoenix: Safe GitHub Issue Resolution via Multi-Agent LLMs [0.0]
  We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation.<n>Phoenix decomposes the work across six specialized agents.<n>A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation.
  arXiv  Detail & Related papers  (2026-06-18T13:56:12Z)
• Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development [0.0]
  Agentic AI coding systems can inspect repositories, plan implementation steps, edit files, run tests, and submit pull requests.<n>Current evidence does not support the simple claim that autonomous code generation automatically improves engineering outcomes.<n>This paper argues that the central problem is no longer prompt engineering; it is engineering process control.
  arXiv  Detail & Related papers  (2026-05-19T20:10:13Z)
• SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair [1.6922906233636834]
  Architectural code smells erode software maintainability and are costly to repair manually.<n>We present the first empirical evaluation of large language model agents on architectural code smell repair.<n>We contribute SmellBench, a task orchestration framework that incorporates smell-type-specific optimized prompts.
  arXiv  Detail & Related papers  (2026-05-07T22:33:32Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Pull Requests as a Training Signal for Repo-Level Code Editing [49.82435173554125]
  Clean Pull Request (Clean-PR) is a mid-training paradigm that leverages real-world GitHub pull requests as a training signal for repository-level editing.<n>We introduce a scalable pipeline that converts noisy pull request diffs into Search/Replace edit blocks through reconstruction and validation.<n>On SWE-bench, our model significantly outperforms the instruction-tuned baseline, achieving absolute improvements of 13.6% on SWE-bench Lite and 12.3% on SWE-bench Verified.
  arXiv  Detail & Related papers  (2026-02-07T09:22:25Z)
• VibeTensor: System Software for Deep Learning, Fully Generated by AI Agents [42.56489784841984]
  "fully generated" refers to code provenance: implementation changes were produced and applied as agent-proposed diffs.<n>We describe the architecture, summarize the workflow used to produce and validate the system, and evaluate the artifact.
  arXiv  Detail & Related papers  (2026-01-21T19:29:00Z)
• InfCode: Adversarial Iterative Refinement of Tests and Patches for Reliable Software Issue Resolution [31.874379525390967]
  InfCode is an adversarial multi-agent framework for automated repository-level issue resolution.<n>InfCode iteratively refines both tests and patches through adversarial interaction between a Test Patch Generator and a Code Patch Generator.<n> Experiments on SWE-bench Lite and SWE-bench Verified using models such as DeepSeek-V3 and Claude 4.5 Sonnet show that InfCode consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-11-20T03:04:21Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• A Unified Debugging Approach via LLM-Based Multi-Agent Synergy [39.11825182386288]
  FixAgent is an end-to-end framework for unified debug through multi-agent synergy. It significantly outperforms state-of-the-art repair methods, fixing 1.25$times$ to 2.56$times$ bugs on the repo-level benchmark, Defects4J.
  arXiv  Detail & Related papers  (2024-04-26T04:55:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.