LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World

• URL: http://arxiv.org/abs/2502.02539v1
• Date: Tue, 04 Feb 2025 18:06:04 GMT
• Title: LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World
• Authors: Shrikara Arun, Meghana Tedla, Karthik Vaidhyanathan,
• Abstract summary: This paper studies the capability of Large Language Models to generate architectural components for Functions as a Service (F)<n>The small size of their architectural components make this architectural style amenable for generation using current LLMs.<n>We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, the exponential growth in capability and pervasiveness of Large Language Models (LLMs) has led to significant work done in the field of code generation. However, this generation has been limited to code snippets. Going one step further, our desideratum is to automatically generate architectural components. This would not only speed up development time, but would also enable us to eventually completely skip the development phase, moving directly from design decisions to deployment. To this end, we conduct an exploratory study on the capability of LLMs to generate architectural components for Functions as a Service (FaaS), commonly known as serverless functions. The small size of their architectural components make this architectural style amenable for generation using current LLMs compared to other styles like monoliths and microservices. We perform the study by systematically selecting open source serverless repositories, masking a serverless function and utilizing state of the art LLMs provided with varying levels of context information about the overall system to generate the masked function. We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains to evaluate code quality and the degree of similarity between human and LLM generated code respectively. Along with our findings, we also present a discussion on the path forward for using GenAI in architectural component generation.

Related papers

• FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation [26.14778133391999]
  FEA-Bench is a benchmark designed to assess the ability of large language models to perform incremental development within code repositories. We collect pull requests from 83 GitHub repositories and use rule-based and intent-based filtering to construct task instances focused on new feature development.
  arXiv  Detail & Related papers  (2025-03-09T16:11:57Z)
• SnipGen: A Mining Repository Framework for Evaluating LLMs for Code [51.07471575337676]
  Language Models (LLMs) are trained on extensive datasets that include code repositories. evaluating their effectiveness poses significant challenges due to the potential overlap between the datasets used for training and those employed for evaluation. We introduce SnipGen, a comprehensive repository mining framework designed to leverage prompt engineering across various downstream tasks for code generation.
  arXiv  Detail & Related papers  (2025-02-10T21:28:15Z)
• HiVeGen -- Hierarchical LLM-based Verilog Generation for Scalable Chip Design [55.54477725000291]
  HiVeGen is a hierarchical Verilog generation framework that decomposes generation tasks into hierarchical submodules.<n> automatic Design Space Exploration (DSE) into hierarchy-aware prompt generation, introducing weight-based retrieval to enhance code reuse.<n>Real-time human-computer interaction to lower error-correction cost, significantly improving the quality of generated designs.
  arXiv  Detail & Related papers  (2024-12-06T19:37:53Z)
• Specifications: The missing link to making the development of LLM systems an engineering discipline [65.10077876035417]
  We discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute. We outline several future directions for research to enable the development of modular and reliable LLM-based systems.
  arXiv  Detail & Related papers  (2024-11-25T07:48:31Z)
• The Compressor-Retriever Architecture for Language Model OS [20.56093501980724]
  This paper explores the concept of using a language model as the core component of an operating system (OS) A key challenge in realizing such an LM OS is managing the life-long context and ensuring statefulness across sessions. We introduce compressor-retriever, a model-agnostic architecture designed for life-long context management.
  arXiv  Detail & Related papers  (2024-09-02T23:28:15Z)
• Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models [8.02264001053969]
  Large language models (LLMs) have shown remarkable performance across a wide range of applications, often outperforming human experts. With constant innovation in LLM serving optimizations and model architecture evolving at breakneck speed, the hardware requirements to meet Service Level Objectives (SLOs) remain an open research question. We present an analytical tool, GenZ, to efficiently navigate the relationship between diverse LLM model architectures and AI platform design parameters.
  arXiv  Detail & Related papers  (2024-06-03T18:00:50Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• Evaluating In-Context Learning of Libraries for Code Generation [35.57902679044737]
  Large Language Models (LLMs) exhibit a high degree of code generation and comprehension capability. Recent work has shown that large proprietary LLMs can learn novel library usage in-context from demonstrations.
  arXiv  Detail & Related papers  (2023-11-16T07:37:25Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• CodeTF: One-stop Transformer Library for State-of-the-art Code LLM [72.1638273937025]
  We present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Our library supports a collection of pretrained Code LLM models and popular code benchmarks. We hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering.
  arXiv  Detail & Related papers  (2023-05-31T05:24:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.