DRAFT-ing Architectural Design Decisions using LLMs

• URL: http://arxiv.org/abs/2504.08207v1
• Date: Fri, 11 Apr 2025 02:19:01 GMT
• Title: DRAFT-ing Architectural Design Decisions using LLMs
• Abstract summary: Domain Specific Retreival Augumented Few Shot Fine Tuninng, DRAFT, is a tool for drafting Architecture Design Decisions (ADDs)<n>It combines the strengths of few-shot, retrieval-augmented generation and fine-tuning approaches to generate ADDs.<n>Our findings indicate that DRAFT can aid architects in drafting ADDs while addressing privacy and resource constraints.
• Score: 7.631177373258237
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Architectural Knowledge Management (AKM) is crucial for software development but remains challenging due to the lack of standardization and high manual effort. Architecture Decision Records (ADRs) provide a structured approach to capture Architecture Design Decisions (ADDs), but their adoption is limited due to the manual effort involved and insufficient tool support. Our previous work has shown that Large Language Models (LLMs) can assist in generating ADDs. However, simply prompting the LLM does not produce quality ADDs. Moreover, using third-party LLMs raises privacy concerns, while self-hosting them poses resource challenges. To this end, we experimented with different approaches like few-shot, retrieval-augmented generation (RAG) and fine-tuning to enhance LLM's ability to generate ADDs. Our results show that both techniques improve effectiveness. Building on this, we propose Domain Specific Retreival Augumented Few Shot Fine Tuninng, DRAFT, which combines the strengths of all these three approaches for more effective ADD generation. DRAFT operates in two phases: an offline phase that fine-tunes an LLM on generating ADDs augmented with retrieved examples and an online phase that generates ADDs by leveraging retrieved ADRs and the fine-tuned model. We evaluated DRAFT against existing approaches on a dataset of 4,911 ADRs and various LLMs and analyzed them using automated metrics and human evaluations. Results show DRAFT outperforms all other approaches in effectiveness while maintaining efficiency. Our findings indicate that DRAFT can aid architects in drafting ADDs while addressing privacy and resource constraints.

Related papers

• Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study [0.09332987715848716]
  Architectural Design Decisions (ADDs) capture the rationale behind the structure and evolution of software systems.<n>Extracting ADDs from commits is challenging due to their implicit and unstructured nature.<n>Large Language Models (LLMs) have shown strong capabilities in understanding code and text.
  arXiv  Detail & Related papers  (2026-09-03T11:54:27Z)
• VRAG-DFD: Verifiable Retrieval-Augmentation for MLLM-based Deepfake Detection [49.88962095967614]
  We propose the VRAG-DFD framework with accurate dynamic forgery knowledge retrieval and powerful critical reasoning capabilities.<n>In terms of data, we constructed two datasets with RAG: Forensic Knowledge Database (FKD) for DFD knowledge annotation, and Forensic Chain-of-Thought dataset (F-CoT) for critical CoT construction.<n>In terms of performance, VRAG-DFD achieved SOTA and competitive performance on DFD generalization testing.
  arXiv  Detail & Related papers  (2026-04-15T09:27:49Z)
• How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practices [81.85465545346266]
  Diffusion language models (DLMs) have emerged as a promising alternative to the long-dominant autoregressive (AR) paradigm.<n>Yet, current open-source DLMs often underperform their AR counterparts in speed, limiting their real-world utility.<n>This work presents a systematic study of DLM efficiency, identifying key issues in prior evaluation methods.
  arXiv  Detail & Related papers  (2025-10-21T10:00:32Z)
• An LLM-assisted approach to designing software architectures using ADD [13.527856271380317]
  This paper proposes an approach for Large Language Model (LLM)-assisted software architecture design using the Attribute-Driven Design (ADD) method.<n>We validate the approach through case studies, comparing generated designs against proven solutions and evaluating them with professional architects.
  arXiv  Detail & Related papers  (2025-06-27T23:58:15Z)
• Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions [35.77671870515212]
  The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and dataset Distillation (DD)
  arXiv  Detail & Related papers  (2025-04-20T23:50:23Z)
• Automated Archival Descriptions with Federated Intelligence of LLMs [2.271344459418284]
  This work aims at exploring the potential of agentic AI and large language models (LLMs) in addressing the challenges of implementing a standardized archival description process.<n>We introduce an agentic AI-driven system for automated generation of high-quality metadata descriptions of archival materials.
  arXiv  Detail & Related papers  (2025-04-08T06:11:05Z)
• Island-Based Evolutionary Computation with Diverse Surrogates and Adaptive Knowledge Transfer for High-Dimensional Data-Driven Optimization [22.363265173423454]
  This paper proposes an offline data-driven evolutionary algorithm named DSKT-DDEA.<n>Our algorithm is competitive with state-of-the-art DDEAs on problems with up to 1000 dimensions.
  arXiv  Detail & Related papers  (2025-03-17T06:35:59Z)
• Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models [79.2162092822111]
  We systematically evaluate reinforcement learning (RL) and control-based methods on a suite of navigation tasks.<n>We employ a latent dynamics model using the Joint Embedding Predictive Architecture (JEPA) and employ it for planning.<n>Our results show that model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts.
  arXiv  Detail & Related papers  (2025-02-20T18:39:41Z)
• Scaling Autonomous Agents via Automatic Reward Modeling And Planning [52.39395405893965]
  Large language models (LLMs) have demonstrated remarkable capabilities across a range of tasks. However, they still struggle with problems requiring multi-step decision-making and environmental feedback. We propose a framework that can automatically learn a reward model from the environment without human annotations.
  arXiv  Detail & Related papers  (2025-02-17T18:49:25Z)
• AD-LLM: Benchmarking Large Language Models for Anomaly Detection [50.57641458208208]
  This paper introduces AD-LLM, the first benchmark that evaluates how large language models can help with anomaly detection. We examine three key tasks: zero-shot detection, using LLMs' pre-trained knowledge to perform AD without tasks-specific training; data augmentation, generating synthetic data and category descriptions to improve AD models; and model selection, using LLMs to suggest unsupervised AD models.
  arXiv  Detail & Related papers  (2024-12-15T10:22:14Z)
• Dataset Distillation from First Principles: Integrating Core Information Extraction and Purposeful Learning [10.116674195405126]
  We argue that a precise characterization of the underlying optimization problem must specify the inference task associated with the application of interest. Our formalization reveals novel applications of DD across different modeling environments. We present numerical results for two case studies important in contemporary settings.
  arXiv  Detail & Related papers  (2024-09-02T18:11:15Z)
• Relative Difficulty Distillation for Semantic Segmentation [54.76143187709987]
  We propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD) RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals. Our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.
  arXiv  Detail & Related papers  (2024-07-04T08:08:25Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study [10.051928501109]
  An essential artifact of Architectural Knowledge Management (AKM) is the Architecture Decision Records ( ADRs) Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation.
  arXiv  Detail & Related papers  (2024-03-04T03:56:14Z)
• LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition [67.96794382040547]
  $LLM-DA$ is a novel data augmentation technique based on large language models (LLMs) for the few-shot NER task. Our approach involves employing 14 contextual rewriting strategies, designing entity replacements of the same type, and incorporating noise injection to enhance robustness.
  arXiv  Detail & Related papers  (2024-02-22T14:19:56Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)
• Editing Large Language Models: Problems, Methods, and Opportunities [51.903537096207]
  This paper embarks on a deep exploration of the problems, methods, and opportunities related to model editing for LLMs. We provide an exhaustive overview of the task definition and challenges associated with model editing, along with an in-depth empirical analysis of the most progressive methods currently at our disposal. Our objective is to provide valuable insights into the effectiveness and feasibility of each editing technique, thereby assisting the community in making informed decisions on the selection of the most appropriate method for a specific task or context.
  arXiv  Detail & Related papers  (2023-05-22T16:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.