EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

• URL: http://arxiv.org/abs/2605.19743v2
• Date: Wed, 27 May 2026 12:23:43 GMT
• Title: EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design
• Abstract summary: We introduce a benchmark suite with three evaluation dimensions.<n>We present EngiAI, a Multi-Agent System (MAS) implementation built on LangGraph.
• Score: 0.4499833362998488
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address multi-agent systems that combine simulation, retrieval, and manufacturing preparation. We introduce a benchmark suite with three evaluation dimensions: (1) a workflow benchmark with seven prompt styles targeting distinct cognitive demands-including direct tool use, semantic disambiguation, conditional branching, and working-memory tasks; (2) a Retrieval-Augmented Generation (RAG) benchmark with gated scoring isolating retrieval contributions to parameter selection; and (3) an High Performance Computing (HPC) benchmark evaluating end-to-end ML training orchestration on a SLURM cluster. Alongside the benchmark we present EngiAI, a Multi-Agent System (MAS) reference implementation built on LangGraph that operationalizes the benchmark by coordinating seven specialized agents through a supervisor architecture, unifying topology optimization, document retrieval, HPC job orchestration, and 3D printer control. Across four LLM backends and two EngiBench problems, proprietary models achieve 96-97% average task completion on Beams2D, while open-source 4B-parameter models reach 55-78%, with clear generational improvement. Conditional branching proves most challenging, with task completion dropping to 20-53% for the conditional style on Photonics2D. RAG gating confirms near-perfect retrieval-augmented scores (about 1.0) versus near-zero without retrieval, validating the evaluation design. On HPC orchestration, one model completes all pipeline steps in 100% of runs while another drops to 50%, revealing that multi-step instruction following degrades over long-running workflows.

Related papers

• Memory-Guided Tree Search with Cross-Branch Knowledge Transfer for LLM Solver Synthesis [3.0663322946413287]
  We introduce MEMOIR, a memory-guided tree-search framework with a two-level memory hierarchy.<n>Across seven Combinatorial optimization problems, MEMOIR achieves 96.7% solution validity.<n>MeMOIR's run-to-run validity standard deviation is more than an order of magnitude below that of every baseline we evaluated.
  arXiv  Detail & Related papers  (2026-05-17T16:47:31Z)
• Multi-Paradigm Agent Interaction in Practice:A Systematic Analysis of Generator-Evaluator, ReAct Loop,and Adversarial Evaluation in the buddyMe Framework [17.155004351157093]
  This paper presents a systematic analysis of three principal agent interaction paradigms, including Multi-Agent Orchestration (Generator-Evaluator), ReAct Tool-Use Loops, and Memory-Augmented Interaction, as implemented in buddyMe.
  arXiv  Detail & Related papers  (2026-05-16T05:35:50Z)
• Toward Autonomous SOC Operations: End-to-End LLM Framework for Threat Detection, Query Generation, and Resolution in Security Operations [1.7102338932907293]
  Security Operations Centers (SOCs) face mounting operational challenges.<n>These challenges come from increasing threat volumes, heterogeneous SIEM platforms, and time-consuming manual triage.<n>We present an end-to-end threat management framework that integrates ensemble-based detection, syntax-constrained query generation, and retrieval-augmented resolution support.
  arXiv  Detail & Related papers  (2026-04-30T02:06:46Z)
• Why Search When You Can Transfer? Amortized Agentic Workflow Design from Structural Priors [51.95306551061392]
  Automated agentic workflow design currently relies on per-task iterative search.<n>We present SWIFT, a framework that amortizes workflow design into reusable structural priors.<n> SWIFT outperforms the state-of-the-art search-based method on five benchmarks.
  arXiv  Detail & Related papers  (2026-04-27T21:25:00Z)
• VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation [98.38575149237442]
  VLAA-GUI is a modular GUI framework built around three integrated components.<n>A mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step.<n>A mandatory Loop Breaker provides multi-tier switching interaction mode after repeated failures.
  arXiv  Detail & Related papers  (2026-04-23T07:42:37Z)
• Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration [57.23877089929136]
  Graph-of-Agents (GoA) is a new graph-based framework for modeling multi-agent LLM communication.<n>GoA achieves superior performance using only 3 selected agents, outperforming recent multi-agent LLM baselines.
  arXiv  Detail & Related papers  (2026-04-18T21:13:03Z)
• Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMs [62.17306142810532]
  We present Chimera, a predictive scheduling system for multi-agent workflow serving on heterogeneous LLM clusters.<n>Chime traces the best latency, reducing end-to-end latency by 1.2--2.4$times$ and improving task performance by 8.0-9.5 percentage points on average over competitive baselines.
  arXiv  Detail & Related papers  (2026-03-23T17:01:42Z)
• Unifying Ranking and Generation in Query Auto-Completion via Retrieval-Augmented Generation and Multi-Objective Alignment [8.610245271469267]
  Query Auto-Completion (QAC) suggests query completions as users type, helping them articulate intent and reach results more efficiently.<n>Traditional retrieve-and-rank pipelines have limited long-tail coverage and require extensive feature engineering.<n>We present a unified framework that reformulates QAC as end-to-end list generation through Retrieval-Augmented Generation (RAG) and multi-objective Direct Preference Optimization (DPO)
  arXiv  Detail & Related papers  (2026-02-01T05:15:07Z)
• MortgageLLM: Domain-Adaptive Pretraining with Residual Instruction Transfer, Alignment Tuning, and Task-Specific Routing [0.7367330074083941]
  We present MortgageLLM, a novel domain-specific large language model.<n>It is developed using a dual-track specialization framework from a single base model.<n>We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself.
  arXiv  Detail & Related papers  (2025-11-26T06:37:57Z)
• AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search [58.98450205734779]
  Large language model (LLM) agents have demonstrated strong capabilities across diverse domains.<n>Existing agent search methods suffer from three major limitations.<n>We introduce a comprehensive framework to address these challenges.
  arXiv  Detail & Related papers  (2025-06-06T12:07:23Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.