Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study

• URL: http://arxiv.org/abs/2403.01709v1
• Date: Mon, 4 Mar 2024 03:56:14 GMT
• Title: Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study
• Abstract summary: An essential artifact of Architectural Knowledge Management (AKM) is the Architecture Decision Records ( ADRs) Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation.
• Score: 10.051928501109
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Architectural Knowledge Management (AKM) involves the organized handling of information related to architectural decisions and design within a project or organization. An essential artifact of AKM is the Architecture Decision Records (ADR), which documents key design decisions. ADRs are documents that capture decision context, decision made and various aspects related to a design decision, thereby promoting transparency, collaboration, and understanding. Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation. However, the effectiveness of LLM for ADR generation or understanding is something that has not been explored. To this end, in this work, we perform an exploratory study that aims to investigate the feasibility of using LLM for the generation of ADRs given the decision context. In our exploratory study, we utilize GPT and T5-based models with 0-shot, few-shot, and fine-tuning approaches to generate the Decision of an ADR given its Context. Our results indicate that in a 0-shot setting, state-of-the-art models such as GPT-4 generate relevant and accurate Design Decisions, although they fall short of human-level performance. Additionally, we observe that more cost-effective models like GPT-3.5 can achieve similar outcomes in a few-shot setting, and smaller models such as Flan-T5 can yield comparable results after fine-tuning. To conclude, this exploratory study suggests that LLM can generate Design Decisions, but further research is required to attain human-level generation and establish standardized widespread adoption.

Related papers

• Interaction-Grounded Learning for Contextual Markov Decision Processes with Personalized Feedback [59.287761696290865]
  We propose a computationally efficient algorithm that achieves a sublinear regret guarantee for contextual episodic Markov Decision Processes (MDPs) with personalized feedback.<n>We demonstrate the effectiveness of our method in learning personalized objectives from multi-turn interactions through experiments on both a synthetic episodic MDP and a real-world user booking dataset.
  arXiv  Detail & Related papers  (2026-02-09T06:29:54Z)
• DEVAL: A Framework for Evaluating and Improving the Derivation Capability of Large Language Models [25.789044700689285]
  Human reasoning can derive corresponding modifications to the output based on certain kinds of changes to the input.<n>This reasoning pattern has not been comprehensively described or evaluated in Large Language Models.<n>We propose a novel prompt engineering approach called Derivation Prompting.
  arXiv  Detail & Related papers  (2025-11-18T02:37:27Z)
• How Good are Foundation Models in Step-by-Step Embodied Reasoning? [79.15268080287505]
  Embodied agents must make decisions that are safe, spatially coherent, and grounded in context.<n>Recent advances in large multimodal models have shown promising capabilities in visual understanding and language generation.<n>Our benchmark includes over 1.1k samples with detailed step-by-step reasoning across 10 tasks and 8 embodiments.
  arXiv  Detail & Related papers  (2025-09-18T17:56:30Z)
• Using LLMs in Generating Design Rationale for Software Architecture Decisions [6.314112341291189]
  Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions.
  arXiv  Detail & Related papers  (2025-04-29T14:00:18Z)
• DRAFT-ing Architectural Design Decisions using LLMs [7.631177373258237]
  Domain Specific Retreival Augumented Few Shot Fine Tuninng, DRAFT, is a tool for drafting Architecture Design Decisions (ADDs) It combines the strengths of few-shot, retrieval-augmented generation and fine-tuning approaches to generate ADDs. Our findings indicate that DRAFT can aid architects in drafting ADDs while addressing privacy and resource constraints.
  arXiv  Detail & Related papers  (2025-04-11T02:19:01Z)
• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• EVOLvE: Evaluating and Optimizing LLMs For Exploration [76.66831821738927]
  Large language models (LLMs) remain under-studied in scenarios requiring optimal decision-making under uncertainty. We measure LLMs' (in)ability to make optimal decisions in bandits, a state-less reinforcement learning setting relevant to many applications. Motivated by the existence of optimal exploration algorithms, we propose efficient ways to integrate this algorithmic knowledge into LLMs.
  arXiv  Detail & Related papers  (2024-10-08T17:54:03Z)
• Making Large Language Models Better Planners with Reasoning-Decision Alignment [70.5381163219608]
  We motivate an end-to-end decision-making model based on multimodality-augmented LLM. We propose a reasoning-decision alignment constraint between the paired CoTs and planning results. We dub our proposed large language planners with reasoning-decision alignment as RDA-Driver.
  arXiv  Detail & Related papers  (2024-08-25T16:43:47Z)
• Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making [51.737762570776006]
  LLM-ACTR is a novel neuro-symbolic architecture that provides human-aligned and versatile decision-making. Our framework extracts and embeds knowledge of ACT-R's internal decision-making process as latent neural representations. Our experiments on novel Design for Manufacturing tasks show both improved task performance as well as improved grounded decision-making capability.
  arXiv  Detail & Related papers  (2024-08-17T11:49:53Z)
• CAUS: A Dataset for Question Generation based on Human Cognition Leveraging Large Language Models [4.962252439662465]
  We introduce the Curious About Uncertain Scene dataset to enable Large Language Models to emulate human cognitive processes for resolving uncertainties. Our approach involves providing scene descriptions embedded with uncertainties to stimulate the generation of reasoning and queries. Our results demonstrate that GPT-4 can effectively generate pertinent questions and grasp their nuances, particularly when given appropriate context and instructions.
  arXiv  Detail & Related papers  (2024-04-18T01:31:19Z)
• Learning Planning-based Reasoning by Trajectories Collection and Process Reward Synthesizing [61.98556945939045]
  We propose a framework to learn planning-based reasoning through Direct Preference Optimization (DPO) on collected trajectories. Our results on challenging logical reasoning benchmarks demonstrate the effectiveness of our learning framework.
  arXiv  Detail & Related papers  (2024-02-01T15:18:33Z)
• Rational Decision-Making Agent with Internalized Utility Judgment [91.80700126895927]
  Large language models (LLMs) have demonstrated remarkable advancements and have attracted significant efforts to develop LLMs into agents capable of executing intricate multi-step decision-making tasks beyond traditional NLP applications. This paper proposes RadAgent, which fosters the development of its rationality through an iterative framework involving Experience Exploration and Utility Learning. Experimental results on the ToolBench dataset demonstrate RadAgent's superiority over baselines, achieving over 10% improvement in Pass Rate on diverse tasks.
  arXiv  Detail & Related papers  (2023-08-24T03:11:45Z)
• Scaling Evidence-based Instructional Design Expertise through Large Language Models [0.0]
  This paper explores leveraging Large Language Models (LLMs), specifically GPT-4, in the field of instructional design. With a focus on scaling evidence-based instructional design expertise, our research aims to bridge the gap between theoretical educational studies and practical implementation. We discuss the benefits and limitations of AI-driven content generation, emphasizing the necessity of human oversight in ensuring the quality of educational materials.
  arXiv  Detail & Related papers  (2023-05-31T17:54:07Z)
• Leveraging Expert Consistency to Improve Algorithmic Decision Support [62.61153549123407]
  We explore the use of historical expert decisions as a rich source of information that can be combined with observed outcomes to narrow the construct gap. We propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data is assessed by a single expert. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap.
  arXiv  Detail & Related papers  (2021-01-24T05:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.