Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction

• URL: http://arxiv.org/abs/2406.16903v1
• Date: Sun, 2 Jun 2024 17:47:57 GMT
• Title: Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction
• Authors: Changyu Du, Stavros Nousias, André Borrmann,
• Abstract summary: Designers often seek to interact with the software in a more intelligent and lightweight manner. We propose an autonomous agent framework that can function as a copilot in the BIM authoring tool. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly.
• Score: 0.40964539027092917
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Facing increasingly complex BIM authoring software and the accompanying expensive learning costs, designers often seek to interact with the software in a more intelligent and lightweight manner. They aim to automate modeling workflows, avoiding obstacles and difficulties caused by software usage, thereby focusing on the design process itself. To address this issue, we proposed an LLM-based autonomous agent framework that can function as a copilot in the BIM authoring tool, answering software usage questions, understanding the user's design intentions from natural language, and autonomously executing modeling tasks by invoking the appropriate tools. In a case study based on the BIM authoring software Vectorworks, we implemented a software prototype to integrate the proposed framework seamlessly into the BIM authoring scenario. We evaluated the planning and reasoning capabilities of different LLMs within this framework when faced with complex instructions. Our work demonstrates the significant potential of LLM-based agents in design automation and intelligent interaction.

Related papers

• Adaptive Tool Use in Large Language Models with Meta-Cognition Trigger [49.81945268343162]
  We propose MeCo, an adaptive decision-making strategy for external tool use. MeCo captures high-level cognitive signals in the representation space, guiding when to invoke tools. Our experiments show that MeCo accurately detects LLMs' internal cognitive signals and significantly improves tool-use decision-making.
  arXiv  Detail & Related papers  (2025-02-18T15:45:01Z)
• Every Software as an Agent: Blueprint and Case Study [0.6655461660736298]
  We advocate to endow large language models (LLMs) with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. We present an overall design architecture and case studies on two popular web-based desktop applications.
  arXiv  Detail & Related papers  (2025-02-07T08:29:09Z)
• Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs [0.0]
  Large Language Models (LLMs) have revolutionized various aspects of engineering and science. This thesis serves as a comprehensive guide that elucidates a multi-faceted approach for empowering LLMs with the capability to leverage Application Programming Interfaces (APIs) We propose an on-device architecture that aims to exploit the functionality of carry-on devices by using small models from the Hugging Face community.
  arXiv  Detail & Related papers  (2024-12-17T14:14:04Z)
• Human-In-the-Loop Software Development Agents [12.830816751625829]
  Large Language Models (LLMs)-based multi-agent paradigms for software engineering are introduced to automatically resolve software development tasks. In this paper, we introduce a Human-in-the-loop LLM-based Agents framework (HULA) for software development. We design, implement, and deploy the HULA framework into Atlassian for internal uses.
  arXiv  Detail & Related papers  (2024-11-19T23:22:33Z)
• Learning to Ask: When LLM Agents Meet Unclear Instruction [55.65312637965779]
  Large language models (LLMs) can leverage external tools for addressing a range of tasks unattainable through language skills alone. We evaluate the performance of LLMs tool-use under imperfect instructions, analyze the error patterns, and build a challenging tool-use benchmark called Noisy ToolBench. We propose a novel framework, Ask-when-Needed (AwN), which prompts LLMs to ask questions to users whenever they encounter obstacles due to unclear instructions.
  arXiv  Detail & Related papers  (2024-08-31T23:06:12Z)
• Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Framework [0.3749861135832073]
  Text2 BIM is a multi-agent framework that generates 3D building models from natural language instructions. A rule-based model checker is introduced into the agentic workflow to guide the LLM agents in resolving issues within the generated models. The framework can effectively generate high-quality, structurally rational building models that are aligned with the abstract concepts specified by user input.
  arXiv  Detail & Related papers  (2024-08-15T09:48:45Z)
• WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks [85.95607119635102]
  Large language models (LLMs) can mimic human-like intelligence. WorkArena++ is designed to evaluate the planning, problem-solving, logical/arithmetic reasoning, retrieval, and contextual understanding abilities of web agents.
  arXiv  Detail & Related papers  (2024-07-07T07:15:49Z)
• Meaning-Typed Programming: Language-level Abstractions and Runtime for GenAI Applications [8.308424118055981]
  Software is rapidly evolving from logical code to neuro-integrated applications that leverage generative AI and large language models (LLMs) for application functionality. This paper proposes meaning-typed programming (MTP), a novel approach to simplify the creation of neuro-integrated applications.
  arXiv  Detail & Related papers  (2024-05-14T21:12:01Z)
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering [79.07755560048388]
  SWE-agent is a system that facilitates LM agents to autonomously use computers to solve software engineering tasks. SWE-agent's custom agent-computer interface (ACI) significantly enhances an agent's ability to create and edit code files, navigate entire repositories, and execute tests and other programs. We evaluate SWE-agent on SWE-bench and HumanEvalFix, achieving state-of-the-art performance on both with a pass@1 rate of 12.5% and 87.7%, respectively.
  arXiv  Detail & Related papers  (2024-05-06T17:41:33Z)
• TaskBench: Benchmarking Large Language Models for Task Automation [82.2932794189585]
  We introduce TaskBench, a framework to evaluate the capability of large language models (LLMs) in task automation. Specifically, task decomposition, tool selection, and parameter prediction are assessed. Our approach combines automated construction with rigorous human verification, ensuring high consistency with human evaluation.
  arXiv  Detail & Related papers  (2023-11-30T18:02:44Z)
• Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
  We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering. In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.