Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

• URL: http://arxiv.org/abs/2608.26199v1
• Date: Tue, 25 Aug 2026 15:33:50 GMT
• Title: Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling
• Abstract summary: We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design in an industry-realistic tool-calling setting.<n>To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development.<n>We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures.
• Score: 1.6639892342953113
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations---such as creating components, adding ports, and wiring connections---through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.

Related papers

• LLM-based Hardware Development with Hierarchical IRs and End-to-End Multi-Agent Workflow [3.3126072964592836]
  Large language models (LLMs) are increasingly used in software development, but their use in complex hardware design remains limited.<n>We present an LLM-based hardware development framework with hierarchical intermediate representations (IRs) and an end-to-end multi-agent workflow.<n>We evaluate our framework on Verilog-Eval benchmark, achieving a pass@5 rate of 95.5%, which surpasses current state-of-the-art LLM generation frameworks.
  arXiv  Detail & Related papers  (2026-08-31T11:57:37Z)
• ToolRosetta: Bridging Open-Source Repositories and Large Language Model Agents through Automated Tool Standardization [51.92237664440418]
  ToolRosetta is a framework that automatically translates open-source code repositories and APIs into MCP-compatible tools.<n>Given a user task, ToolRosetta autonomously plans toolchains, identifies relevant toolchains, and converts them into executable MCP services.
  arXiv  Detail & Related papers  (2026-03-10T07:19:43Z)
• The Auton Agentic AI Framework [5.410458076724158]
  The field of Artificial Intelligence is undergoing a transition from Generative AI to Agentic AI.<n>This transition exposes a fundamental architectural mismatch: Large Language Models (LLMs) produce unstructured outputs, whereas the backend infrastructure they must control requires deterministic, schema-conformant inputs.<n>The present paper describes the Auton Agentic AI Framework, a principled architecture for the creation, creation, and governance of autonomous agent.
  arXiv  Detail & Related papers  (2026-02-27T06:42:08Z)
• A Lightweight Modular Framework for Constructing Autonomous Agents Driven by Large Language Models: Design, Implementation, and Applications in AgentForge [1.932555230783329]
  Lightweight, open-source Python framework designed to democratize the construction of LLM-driven autonomous agents.<n>AgentForge introduces three key innovations: (1) a composable skill abstraction that enables fine-grained task decomposition with formally defined input-output contracts, (2) a unified backend interface supporting seamless switching between cloud-based APIs and local inference engines, and (3) a declarative YAML-based configuration system that separates agent logic from implementation details.
  arXiv  Detail & Related papers  (2026-01-19T20:33:26Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• Orchestral AI: A Framework for Agent Orchestration [45.946776875141666]
  Orchestral is a lightweight Python framework that provides a unified, type-safe interface for building LLM agents across major providers.<n>It operates seamlessly across providers, eliminating manual format translation and reducing framework-induced complexity.<n>It supports advanced agent capabilities found in larger frameworks, including rich tool calling, context compaction, sandboxing, user approval, sub-agents, memory management, and MCP integration.
  arXiv  Detail & Related papers  (2026-01-05T22:02:11Z)
• Monadic Context Engineering [59.95390010097654]
  This paper introduces Monadic Context Engineering (MCE) to provide a formal foundation for agent design.<n>We demonstrate how Monads enable robust composition, how Applicatives provide a principled structure for parallel execution, and crucially, how Monad Transformers allow for the systematic composition of these capabilities.<n>This layered approach enables developers to construct complex, resilient, and efficient AI agents from simple, independently verifiable components.
  arXiv  Detail & Related papers  (2025-12-27T01:52:06Z)
• DeepAgent: A General Reasoning Agent with Scalable Toolsets [111.6384541877723]
  DeepAgent is an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution.<n>To address the challenges of long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories.<n>We develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens.
  arXiv  Detail & Related papers  (2025-10-24T16:24:01Z)
• FABRIC: Framework for Agent-Based Realistic Intelligence Creation [3.940391073007047]
  Large language models (LLMs) are increasingly deployed as agents, expected to decompose goals, invoke tools, and verify results in dynamic environments.<n>We present a unified framework for synthesizing agentic data using only LLMs, without any human-in-the-loop supervision.
  arXiv  Detail & Related papers  (2025-10-20T18:20:22Z)
• A Framework for Testing and Adapting REST APIs as LLM Tools [11.757827071584737]
  Large Language Models (LLMs) are increasingly used to build autonomous agents that perform complex tasks with external tools.<n>Current benchmarks overlook these challenges, leaving a gap in assessing API readiness for agent-driven automation.<n>We present a testing framework that systematically evaluates enterprise APIs when wrapped as Python tools for LLM-based agents.
  arXiv  Detail & Related papers  (2025-04-22T02:52:08Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.