Prompting for Performance: Exploring LLMs for Configuring Software

• URL: http://arxiv.org/abs/2507.09790v1
• Date: Sun, 13 Jul 2025 21:05:01 GMT
• Title: Prompting for Performance: Exploring LLMs for Configuring Software
• Authors: Helge Spieker, Théo Matricon, Nassim Belmecheri, Jørn Eirik Betten, Gauthier Le Bartz Lyan, Heraldo Borges, Quentin Mazouni, Dennis Gross, Arnaud Gotlieb, Mathieu Acher,
• Abstract summary: Large language models (LLMs) can assist in performance-oriented software configuration through prompts.<n>We evaluate several LLMs on tasks including identifying relevant options, ranking configurations, and recommending performant configurations across various systems.
• Score: 10.764205218438759
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Software systems usually provide numerous configuration options that can affect performance metrics such as execution time, memory usage, binary size, or bitrate. On the one hand, making informed decisions is challenging and requires domain expertise in options and their combinations. On the other hand, machine learning techniques can search vast configuration spaces, but with a high computational cost, since concrete executions of numerous configurations are required. In this exploratory study, we investigate whether large language models (LLMs) can assist in performance-oriented software configuration through prompts. We evaluate several LLMs on tasks including identifying relevant options, ranking configurations, and recommending performant configurations across various configurable systems, such as compilers, video encoders, and SAT solvers. Our preliminary results reveal both positive abilities and notable limitations: depending on the task and systems, LLMs can well align with expert knowledge, whereas hallucinations or superficial reasoning can emerge in other cases. These findings represent a first step toward systematic evaluations and the design of LLM-based solutions to assist with software configuration.

Related papers

• Oracular Programming: A Modular Foundation for Building LLM-Enabled Software [5.294604210205507]
  Large Language Models have proved surprisingly effective at solving a wide range of tasks from just a handful of examples.<n>Their lack of reliability and modularity limits their capacity to tackle large problems that require many steps of reasoning.<n>We propose oracular programming, a foundational paradigm for building LLM-enabled applications that lets domain experts express high-level problem-solving strategies.
  arXiv  Detail & Related papers  (2025-02-07T20:24:43Z)
• LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators [1.1028525384019312]
  Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks.
  arXiv  Detail & Related papers  (2024-10-31T18:34:59Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)
• Efficient Prompting for LLM-based Generative Internet of Things [88.84327500311464]
  Large language models (LLMs) have demonstrated remarkable capacities on various tasks, and integrating the capacities of LLMs into the Internet of Things (IoT) applications has drawn much research attention recently. Due to security concerns, many institutions avoid accessing state-of-the-art commercial LLM services, requiring the deployment and utilization of open-source LLMs in a local network setting. We propose a LLM-based Generative IoT (GIoT) system deployed in the local network setting in this study.
  arXiv  Detail & Related papers  (2024-06-14T19:24:00Z)
• MouSi: Poly-Visual-Expert Vision-Language Models [132.58949014605477]
  This paper proposes the use of ensemble experts technique to synergize the capabilities of individual visual encoders. This technique introduces a fusion network to unify the processing of outputs from different visual experts. In our implementation, this technique significantly reduces the positional occupancy in models like SAM, from a substantial 4096 to a more efficient and manageable 64 or even down to 1.
  arXiv  Detail & Related papers  (2024-01-30T18:09:11Z)
• Can LLMs Configure Software Tools [0.76146285961466]
  In software engineering, the meticulous configuration of software tools is crucial in ensuring optimal performance within intricate systems. In this study, we embark on an exploration of leveraging Large-Language Models (LLMs) to streamline the software configuration process. Our work presents a novel approach that employs LLMs, such as Chat-GPT, to identify starting conditions and narrow down the search space, improving configuration efficiency.
  arXiv  Detail & Related papers  (2023-12-11T05:03:02Z)
• Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models [26.2566707495948]
  Large Language Models (LLMs) have seen great advance in both academia and industry. We benchmark the end-to-end performance of pre-training, fine-tuning, and serving LLMs in different sizes. Then, we dive deeper to provide a detailed runtime analysis of the sub-modules, including computing and communication operators in LLMs.
  arXiv  Detail & Related papers  (2023-11-07T03:25:56Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)
• Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
  We make a systematic review of the literature, including the general methodology of supervised fine-tuning (SFT)<n>We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies.
  arXiv  Detail & Related papers  (2023-08-21T15:35:16Z)
• CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models [74.22729793816451]
  Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability. We propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems.
  arXiv  Detail & Related papers  (2023-05-23T17:51:52Z)
• LONViZ: Unboxing the black-box of Configurable Software Systems from a Complex Networks Perspective [9.770775293243934]
  This paper proposes a tool, dubbed LONViZ, to facilitate the exploratory analysis of black-boxconfigured software systems. In experiments, we choose four widely used real-world software systems to develop benchmark platforms under 42 different running environments. We find that LONViZ enables both qualitative and quantitative analysis and disclose various interesting hidden patterns and properties of different software systems.
  arXiv  Detail & Related papers  (2022-01-05T03:14:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.