BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling

• URL: http://arxiv.org/abs/2606.20146v3
• Date: Tue, 23 Jun 2026 06:59:35 GMT
• Title: BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling
• Abstract summary: Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts.<n>We introduce BIM-Edit, a benchmark for evaluating LLMs on natural-language editing of Building Information Models.<n>We evaluate outputs along three dimensions: geometric accuracy, semantic validity, and topological consistency.
• Score: 10.272570186591482
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts from textual instructions. In engineering practice, this requires more than creating new geometry, models must also understand existing scenes, edit them correctly, and preserve semantics and relations. However, many CAD benchmarks focus on creating new models rather than editing existing ones, and mostly evaluate geometric correctness. We introduce BIM-Edit, a benchmark for evaluating LLMs on natural-language editing of Building Information Models (BIM) represented in the Industry Foundation Classes (IFC) format. BIM provides a challenging testbed because building models encode geometry together with semantic and relational structure. BIM-Edit contains 324 editing tasks spanning 11 realistic building models and 36 synthetic scenes. Tasks are expressed using three instruction categories - direct, spatial, and topological - covering both explicit and scene-grounded edits. We evaluate outputs along three dimensions: geometric accuracy, semantic validity, and topological consistency. Across evaluated LLMs, the best-performing model achieves only 49.5% average score across the three metrics, and no model fully solves more than 3.4% of tasks. These results demonstrate a substantial gap between current LLM capabilities and the requirements of structured engineering design workflows.

Related papers

• MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation [17.467056775887944]
  MUSE is a Text-to-CAD benchmark focused on complex, editable boundary representation assemblies.<n>MUSE evaluates generated models through a three-stage protocol: code check, geometric check, and design-intent alignment.<n> Experiments on closed-source and open-source LLMs reveal a clear failure cascade from executable code to valid geometry.
  arXiv  Detail & Related papers  (2026-05-27T15:01:59Z)
• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios [58.34124792457706]
  The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
  arXiv  Detail & Related papers  (2026-04-08T12:23:27Z)
• Factuality Matters: When Image Generation and Editing Meet Structured Visuals [46.627460447235855]
  We construct a large-scale dataset of 1.3 million high-quality structured image pairs.<n>We train a unified model that integrates a VLM with FLUX.1 Kontext.<n>A three-stage training curriculum enables progressive feature alignment, knowledge infusion, and reasoning-augmented generation.
  arXiv  Detail & Related papers  (2025-10-06T17:56:55Z)
• Human-in-the-Loop: Quantitative Evaluation of 3D Models Generation by Large Language Models [0.0]
  This paper introduces a human in the loop framework for the quantitative evaluation of Large Language Models generated 3D models.<n>We propose a comprehensive suite of similarity and complexity metrics, including volumetric accuracy, surface alignment, dimensional fidelity, and topological intricacy.<n>Our findings demonstrate improved generation fidelity with increased semantic richness, with code level prompts achieving perfect reconstruction.
  arXiv  Detail & Related papers  (2025-09-06T11:04:15Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Evaluating Large Language Models for Real-World Engineering Tasks [75.97299249823972]
  This paper introduces a curated database comprising over 100 questions derived from authentic, production-oriented engineering scenarios.<n>Using this dataset, we evaluate four state-of-the-art Large Language Models (LLMs)<n>Our results show that LLMs demonstrate strengths in basic temporal and structural reasoning but struggle significantly with abstract reasoning, formal modeling, and context-sensitive engineering logic.
  arXiv  Detail & Related papers  (2025-05-12T14:05:23Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Framework [0.3749861135832073]
  The Text2 BIM framework generates 3D building models from natural language instructions.<n>A rule-based model checker is introduced into the agentic workflow to guide the LLM agents in resolving issues.<n>The framework can effectively generate high-quality, structurally rational building models.
  arXiv  Detail & Related papers  (2024-08-15T09:48:45Z)
• StructLM: Towards Building Generalist Models for Structured Knowledge Grounding [49.10029030628653]
  Large language models' (LLMs) ability to process structured data lags behind state-of-the-art (SoTA) model by an average of 35%. We train a series of models, referred to as StructLM, based on the Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks.
  arXiv  Detail & Related papers  (2024-02-26T15:47:01Z)
• LLM Augmented LLMs: Expanding Capabilities through Composition [56.40953749310957]
  CALM -- Composition to Augment Language Models -- introduces cross-attention between models to compose their representations and enable new capabilities. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13% on tasks like translation into English. When PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40% over the base model for code generation and explanation tasks.
  arXiv  Detail & Related papers  (2024-01-04T18:53:01Z)
• 3D-GPT: Procedural 3D Modeling with Large Language Models [47.72968643115063]
  We introduce 3D-GPT, a framework utilizing large language models(LLMs) for instruction-driven 3D modeling. 3D-GPT positions LLMs as proficient problem solvers, dissecting the procedural 3D modeling tasks into accessible segments and appointing the apt agent for each task. Our empirical investigations confirm that 3D-GPT not only interprets and executes instructions, delivering reliable results but also collaborates effectively with human designers.
  arXiv  Detail & Related papers  (2023-10-19T17:41:48Z)
• An ontology-aided, natural language-based approach for multi-constraint BIM model querying [0.0]
  This paper presents a novel ontology-aided semantic to automatically map natural language queries (NLQs) that contain different constraints into computer-readable codes for querying complex BIM models. A case study about the design-checking of a real-world residential building demonstrates the practical value of the proposed approach in the construction industry.
  arXiv  Detail & Related papers  (2023-03-27T11:35:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.