LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World

• URL: http://arxiv.org/abs/2502.02539v1
• Date: Tue, 04 Feb 2025 18:06:04 GMT
• Title: LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World
• Authors: Shrikara Arun, Meghana Tedla, Karthik Vaidhyanathan,
• Abstract summary: This paper studies the capability of Large Language Models to generate architectural components for Functions as a Service (F) The small size of their architectural components make this architectural style amenable for generation using current LLMs. We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains.
• Score: 0.0
• License:
• Abstract: Recently, the exponential growth in capability and pervasiveness of Large Language Models (LLMs) has led to significant work done in the field of code generation. However, this generation has been limited to code snippets. Going one step further, our desideratum is to automatically generate architectural components. This would not only speed up development time, but would also enable us to eventually completely skip the development phase, moving directly from design decisions to deployment. To this end, we conduct an exploratory study on the capability of LLMs to generate architectural components for Functions as a Service (FaaS), commonly known as serverless functions. The small size of their architectural components make this architectural style amenable for generation using current LLMs compared to other styles like monoliths and microservices. We perform the study by systematically selecting open source serverless repositories, masking a serverless function and utilizing state of the art LLMs provided with varying levels of context information about the overall system to generate the masked function. We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains to evaluate code quality and the degree of similarity between human and LLM generated code respectively. Along with our findings, we also present a discussion on the path forward for using GenAI in architectural component generation.

Related papers

• SnipGen: A Mining Repository Framework for Evaluating LLMs for Code [51.07471575337676]
  Language Models (LLMs) are trained on extensive datasets that include code repositories. evaluating their effectiveness poses significant challenges due to the potential overlap between the datasets used for training and those employed for evaluation. We introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation.
  arXiv  Detail & Related papers  (2025-02-10T21:28:15Z)
• HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Design [55.54477725000291]
  HiVeGen is a hierarchical Verilog generation framework that decomposes generation tasks into hierarchical submodules. automatic Design Space Exploration (DSE) into hierarchy-aware prompt generation, introducing weight-based retrieval to enhance code reuse. Real-time human-computer interaction to lower error-correction cost, significantly improving the quality of generated designs.
  arXiv  Detail & Related papers  (2024-12-06T19:37:53Z)
• Specifications: The missing link to making the development of LLM systems an engineering discipline [65.10077876035417]
  We discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute. We outline several future directions for research to enable the development of modular and reliable LLM-based systems.
  arXiv  Detail & Related papers  (2024-11-25T07:48:31Z)
• The Compressor-Retriever Architecture for Language Model OS [20.56093501980724]
  This paper explores the concept of using a language model as the core component of an operating system (OS) A key challenge in realizing such an LM OS is managing the life-long context and ensuring statefulness across sessions. We introduce compressor-retriever, a model-agnostic architecture designed for life-long context management.
  arXiv  Detail & Related papers  (2024-09-02T23:28:15Z)
• Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository [4.767858874370881]
  We introduce RepoClassBench, a benchmark designed to rigorously evaluate LLMs in generating class-level code within real-world repositories. RepoClassBench includes "Natural Language to Class generation" tasks across Java, Python & C# from a selection of repositories. We introduce Retrieve-Repotools-Reflect (RRR), a novel approach that equips LLMs with static analysis tools to iteratively navigate & reason about repository-level context.
  arXiv  Detail & Related papers  (2024-04-22T03:52:54Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey [18.930417261395906]
  Transformer-based Large Language Models (LLMs) have been applied in diverse areas such as knowledge bases, human interfaces, and dynamic agents. This article offers a survey of the recent advancement in Transformer-based LLM architectures aimed at enhancing the long-context capabilities of LLMs.
  arXiv  Detail & Related papers  (2023-11-21T04:59:17Z)
• Evaluating In-Context Learning of Libraries for Code Generation [35.57902679044737]
  Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations.
  arXiv  Detail & Related papers  (2023-11-16T07:37:25Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• CodeTF: One-stop Transformer Library for State-of-the-art Code LLM [72.1638273937025]
  We present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Our library supports a collection of pretrained Code LLM models and popular code benchmarks. We hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering.
  arXiv  Detail & Related papers  (2023-05-31T05:24:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.