Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction

• URL: http://arxiv.org/abs/2406.16903v1
• Date: Sun, 2 Jun 2024 17:47:57 GMT
• Title: Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction
• Authors: Changyu Du, Stavros Nousias, André Borrmann,
• Abstract summary: Designers often seek to interact with the software in a more intelligent and lightweight manner. We propose an autonomous agent framework that can function as a copilot in the BIM authoring tool. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly.
• Score: 0.40964539027092917
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Facing increasingly complex BIM authoring software and the accompanying expensive learning costs, designers often seek to interact with the software in a more intelligent and lightweight manner. They aim to automate modeling workflows, avoiding obstacles and difficulties caused by software usage, thereby focusing on the design process itself. To address this issue, we proposed an LLM-based autonomous agent framework that can function as a copilot in the BIM authoring tool, answering software usage questions, understanding the user's design intentions from natural language, and autonomously executing modeling tasks by invoking the appropriate tools. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly into the BIM authoring scenario. We evaluated the planning and reasoning capabilities of different LLMs within this framework when faced with complex instructions. Our work demonstrates the significant potential of LLM-based agents in design automation and intelligent interaction.

Related papers

• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• Agentless: Demystifying LLM-based Software Engineering Agents [12.19683999553113]
  We build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic two-phase process of localization followed by repair. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (27.33%) and lowest cost ($0.34) compared with all existing open-source software agents!
  arXiv  Detail & Related papers  (2024-07-01T17:24:45Z)
• Agent-Driven Automatic Software Improvement [55.2480439325792]
  This research proposal aims to explore innovative solutions by focusing on the deployment of agents powered by Large Language Models (LLMs) The iterative nature of agents, which allows for continuous learning and adaptation, can help surpass common challenges in code generation. We aim to use the iterative feedback in these systems to further fine-tune the LLMs underlying the agents, becoming better aligned to the task of automated software improvement.
  arXiv  Detail & Related papers  (2024-06-24T15:45:22Z)
• Position: A Call to Action for a Human-Centered AutoML Paradigm [83.78883610871867]
  Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) We argue that a key to unlocking AutoML's full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems.
  arXiv  Detail & Related papers  (2024-06-05T15:05:24Z)
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering [79.07755560048388]
  SWE-agent is a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively.
  arXiv  Detail & Related papers  (2024-05-06T17:41:33Z)
• LLMind: Orchestrating AI and IoT with LLM for Complex Task Execution [20.186752447895994]
  We present LLMind, an AI agent framework that enables effective collaboration among IoT devices for executing complex tasks. Inspired by the functional specialization theory of the brain, our framework integrates an LLM with domain-specific AI modules, enhancing its capabilities.
  arXiv  Detail & Related papers  (2023-12-14T14:57:58Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [85.3879908356586]
  We introduce TaskBench to evaluate the capability of large language models in task automation. To generate high-quality evaluation datasets, we introduce the concept of Tool Graph. We also propose TaskEval to evaluate the capability of LLMs from different aspects, including task decomposition, tool invocation, and parameter prediction.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage [28.554981886052953]
  Large Language Models (LLMs) have emerged as powerful tools for various real-world applications. Despite their prowess, intrinsic generative abilities of LLMs may prove insufficient for handling complex tasks. This paper proposes a structured framework tailored for LLM-based AI Agents.
  arXiv  Detail & Related papers  (2023-08-07T09:22:03Z)
• Assessing the Use of AutoML for Data-Driven Software Engineering [10.40771687966477]
  AutoML promises to automate the building of end-to-end AI/ML pipelines. Despite the growing interest and high expectations, there is a dearth of information about the extent to which AutoML is currently adopted.
  arXiv  Detail & Related papers  (2023-07-20T11:14:24Z)
• Approach Intelligent Writing Assistants Usability with Seven Stages of Action [9.378355457555319]
  We adopt Norman's seven stages of action as a framework to approach the interaction design of intelligent writing assistants. We illustrate the framework's applicability to writing tasks by providing an example of software tutorial authoring.
  arXiv  Detail & Related papers  (2023-04-06T02:11:55Z)
• Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
  We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering. In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.