Specifications: The missing link to making the development of LLM systems an engineering discipline

• URL: http://arxiv.org/abs/2412.05299v2
• Date: Mon, 16 Dec 2024 08:17:09 GMT
• Title: Specifications: The missing link to making the development of LLM systems an engineering discipline
• Authors: Ion Stoica, Matei Zaharia, Joseph Gonzalez, Ken Goldberg, Koushik Sen, Hao Zhang, Anastasios Angelopoulos, Shishir G. Patil, Lingjiao Chen, Wei-Lin Chiang, Jared Q. Davis,
• Abstract summary: We discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute. We outline several future directions for research to enable the development of modular and reliable LLM-based systems.
• Score: 65.10077876035417
• License:
• Abstract: Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.

Related papers

• Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis [49.998130983414924]
  Large language models (LLMs) can be employed for programming languages such as Python and C++. This paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design.
  arXiv  Detail & Related papers  (2025-02-19T17:53:59Z)
• LLMs for Generation of Architectural Components: An Exploratory Empirical Study in the Serverless World [0.0]
  This paper studies the capability of Large Language Models to generate architectural components for Functions as a Service (F) The small size of their architectural components make this architectural style amenable for generation using current LLMs. We evaluate correctness through existing tests present in the repositories and use metrics from the Software Engineering (SE) and Natural Language Processing (NLP) domains.
  arXiv  Detail & Related papers  (2025-02-04T18:06:04Z)
• A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models [16.250856588632637]
  The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence. These models are increasingly integrated into diverse applications, impacting both research and industry. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models.
  arXiv  Detail & Related papers  (2024-10-08T21:46:52Z)
• Configurable Foundation Models: Building LLMs from a Modular Perspective [115.63847606634268]
  A growing tendency to decompose LLMs into numerous functional modules allows for inference with part of modules and dynamic assembly of modules to tackle complex tasks. We coin the term brick to represent each functional module, designating the modularized structure as customizable foundation models. We present four brick-oriented operations: retrieval and routing, merging, updating, and growing. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions.
  arXiv  Detail & Related papers  (2024-09-04T17:01:02Z)
• CodeTF: One-stop Transformer Library for State-of-the-art Code LLM [72.1638273937025]
  We present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Our library supports a collection of pretrained Code LLM models and popular code benchmarks. We hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering.
  arXiv  Detail & Related papers  (2023-05-31T05:24:48Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)
• Technology Readiness Levels for AI & ML [79.22051549519989]
  Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results. We propose a proven systems engineering approach for machine learning development and deployment.
  arXiv  Detail & Related papers  (2020-06-21T17:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.