AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction

• URL: http://arxiv.org/abs/2603.29199v1
• Date: Tue, 31 Mar 2026 03:10:28 GMT
• Title: AEC-Bench: A Multimodal Benchmark for Agentic Systems in Architecture, Engineering, and Construction
• Abstract summary: The AEC-Bench is a benchmark for evaluating agentic systems on real-world tasks.<n>This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results.
• Score: 1.1841228164483055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The AEC-Bench is a multimodal benchmark for evaluating agentic systems on real-world tasks in the Architecture, Engineering, and Construction (AEC) domain. The benchmark covers tasks requiring drawing understanding, cross-sheet reasoning, and construction project-level coordination. This report describes the benchmark motivation, dataset taxonomy, evaluation protocol, and baseline results across several domain-specific foundation model harnesses. We use AEC-Bench to identify consistent tools and harness design techniques that uniformly improve performance across foundation models in their own base harnesses, such as Claude Code and Codex. We openly release our benchmark dataset, agent harness, and evaluation code for full replicability at https://github.com/nomic-ai/aec-bench under an Apache 2 license.

Related papers

• BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERP [20.47634814037652]
  We introduce BC-Bench, a benchmark designed to evaluate agentic engineering on real-world tasks in Microsoft Dynamics 365 Business Central.<n>BC-Bench comprises 101 manually curated tasks extracted from two Microsoft-owned production repositories.<n>We evaluate multiple frontier models across two agent harnesses, utilizing multi-run metrics to account for nondeterminism.
  arXiv  Detail & Related papers  (2026-08-21T08:16:21Z)
• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• CocoaBench: Evaluating Unified Digital Agents in the Wild [98.84044600506597]
  We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks.<n>We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones.
  arXiv  Detail & Related papers  (2026-04-13T09:00:10Z)
• Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation [8.867061253116367]
  Large Language Models (LLMs) have demonstrated significant potential in automating software engineering tasks.<n> Generating software architecture designs from requirement documents is a crucial step in software development.<n>R2ABench is a novel benchmark comprising diverse real-world software projects paired with comprehensive Product Requirements Documents (PRDs) and expert-curated PlantUML reference diagrams.
  arXiv  Detail & Related papers  (2026-04-08T04:58:36Z)
• ArchBench: Benchmarking Generative-AI for Software Architecture Tasks [0.0]
  ArchBench is the first unified platform for benchmarking large language models on software architecture tasks.<n>It provides a command-line tool with a standardized pipeline for dataset download, inference with trajectory logging, and automated evaluation.
  arXiv  Detail & Related papers  (2026-03-18T15:26:46Z)
• ArchAgent: Scalable Legacy Software Architecture Recovery with LLMs [44.137226823695066]
  ArchAgent is a scalable agent-based framework that combines static analysis, adaptive code segmentation, and LLM-powered synthesis.<n>It reconstructs multiview, business-aligned architectures from cross-repositorys.<n>ArchAgent introduces scalable diagram generation with contextual pruning and integrates cross-repository data to identify business-critical modules.
  arXiv  Detail & Related papers  (2026-01-19T12:39:05Z)
• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• KGCE: Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models [2.4487691107306655]
  KGCE is a novel benchmarking platform that integrates knowledge base enhancement and a dual-graph evaluation framework.<n>We first constructed a dataset comprising 104 education-related tasks, covering Windows, Android, and cross-platform collaborative tasks.<n>To overcome the execution of existing agents in private-domain tasks, we developed an enhanced agent system incorporating a knowledge base specific to school-specific software.
  arXiv  Detail & Related papers  (2026-01-04T04:39:39Z)
• Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem [90.17610617854247]
  We introduce the Agentic Learning Ecosystem (ALE), a foundational infrastructure that optimize the production pipeline for agentic model.<n>ALE consists of three components: ROLL, a post-training framework for weight optimization; ROCK, a sandbox environment manager for trajectory generation; and iFlow CLI, an agent framework for efficient context engineering.<n>We release ROME, an open-source agent grounded by ALE and trained on over one million trajectories.
  arXiv  Detail & Related papers  (2025-12-31T14:03:39Z)
• Bench-CoE: a Framework for Collaboration of Experts from Benchmark [6.278928823352044]
  Large Language Models (LLMs) are key technologies driving intelligent systems to handle multiple tasks.<n>This paper proposes the Bench-CoE framework, which enables Collaboration of Experts (CoE) by effectively leveraging benchmark evaluations.
  arXiv  Detail & Related papers  (2024-12-05T14:03:41Z)
• BenchAgents: Multi-Agent Systems for Structured Benchmark Creation [23.653678381444276]
  BenchAgents is a framework that automates the creation of evaluation benchmarks.<n>We use BenchAgents to create benchmarks to evaluate capabilities related to planning, constraint satisfaction, and causal reasoning.<n>We then use these benchmarks to study state-of-the-art models and extract new insights into common failure modes and model differences.
  arXiv  Detail & Related papers  (2024-10-29T22:56:18Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks.<n>Our framework supports multiple devices and can be easily extended to any environment with a Python interface.<n>The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.