Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction

• URL: http://arxiv.org/abs/2406.16903v1
• Date: Sun, 2 Jun 2024 17:47:57 GMT
• Title: Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction
• Authors: Changyu Du, Stavros Nousias, André Borrmann,
• Abstract summary: Designers often seek to interact with the software in a more intelligent and lightweight manner. We propose an autonomous agent framework that can function as a copilot in the BIM authoring tool. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly.
• Score: 0.40964539027092917
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Facing increasingly complex BIM authoring software and the accompanying expensive learning costs, designers often seek to interact with the software in a more intelligent and lightweight manner. They aim to automate modeling workflows, avoiding obstacles and difficulties caused by software usage, thereby focusing on the design process itself. To address this issue, we proposed an LLM-based autonomous agent framework that can function as a copilot in the BIM authoring tool, answering software usage questions, understanding the user's design intentions from natural language, and autonomously executing modeling tasks by invoking the appropriate tools. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly into the BIM authoring scenario. We evaluated the planning and reasoning capabilities of different LLMs within this framework when faced with complex instructions. Our work demonstrates the significant potential of LLM-based agents in design automation and intelligent interaction.

Related papers

• Learning to Ask: When LLMs Meet Unclear Instruction [49.256630152684764]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• Knowledge Graph Modeling-Driven Large Language Model Operating System (LLM OS) for Task Automation in Process Engineering Problem-Solving [0.0]
  We present the Process Engineering Operations Assistant (PEOA), an AI-driven framework designed to solve complex problems in the chemical and process industries. The framework employs a modular architecture orchestrated by a meta-agent, which serves as the central coordinator. The results demonstrate the framework effectiveness in automating calculations, accelerating prototyping, and providing AI-augmented decision support for industrial processes.
  arXiv  Detail & Related papers  (2024-08-23T13:52:47Z)
• Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Framework [0.3749861135832073]
  Text2 BIM is a multi-agent framework that generates 3D building models from natural language instructions. A rule-based model checker is introduced into the agentic workflow to guide the LLM agents in resolving issues within the generated models. The framework can effectively generate high-quality, structurally rational building models that are aligned with the abstract concepts specified by user input.
  arXiv  Detail & Related papers  (2024-08-15T09:48:45Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• Agent-Driven Automatic Software Improvement [55.2480439325792]
  This research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs) The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation. We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement.
  arXiv  Detail & Related papers  (2024-06-24T15:45:22Z)
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering [79.07755560048388]
  SWE-agent is a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively.
  arXiv  Detail & Related papers  (2024-05-06T17:41:33Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage [28.554981886052953]
  Large Language Models (LLMs) have emerged as powerful tools for various real-world applications. Despite their prowess, intrinsic generative abilities of LLMs may prove insufficient for handling complex tasks. This paper proposes a structured framework tailored for LLM-based AI Agents.
  arXiv  Detail & Related papers  (2023-08-07T09:22:03Z)
• Assessing the Use of AutoML for Data-Driven Software Engineering [10.40771687966477]
  AutoML promises to automate the building of end-to-end AI/ML pipelines. Despite the growing interest and high expectations, there is a dearth of information about the extent to which AutoML is currently adopted.
  arXiv  Detail & Related papers  (2023-07-20T11:14:24Z)
• Approach Intelligent Writing Assistants Usability with Seven Stages of Action [9.378355457555319]
  We adopt Norman's seven stages of action as a framework to approach the interaction design of intelligent writing assistants. We illustrate the framework's applicability to writing tasks by providing an example of software tutorial authoring.
  arXiv  Detail & Related papers  (2023-04-06T02:11:55Z)
• Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
  We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering. In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.