Towards Online Code Specialization of Systems

• URL: http://arxiv.org/abs/2501.11366v1
• Date: Mon, 20 Jan 2025 09:55:15 GMT
• Title: Towards Online Code Specialization of Systems
• Abstract summary: We advocate for a radically different approach for performance-critical low-level systems.<n>We leverage just-in-time compilation to change systems code based on developer-specified specialization points.<n>We demonstrate that online specialization (i) is feasible even for low-level systems code, such as network stacks, (ii) improves system performance without the need for complex cost models, (iii) incurs low developer effort, especially compared to manual exploration.
• Score: 1.470243173272826
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Specializing low-level systems to specifics of the workload they serve and platform they are running on often significantly improves performance. However, specializing systems is difficult because of three compounding challenges: i) specialization for optimal performance requires in-depth compile-time changes; ii) the right combination of specialization choices for optimal performance is hard to predict a priori; and iii) workloads and platform details often change online. In practice, benefits of specialization are thus not attainable for many low-level systems. To address this, we advocate for a radically different approach for performance-critical low-level systems: designing and implementing systems with and for runtime code specialization. We leverage just-in-time compilation to change systems code based on developer-specified specialization points as the system runs. The JIT runtime automatically tries out specialization choices and measures their impact on system performance, e.g. request latency or throughput, to guide the search. With Iridescent, our early prototype, we demonstrate that online specialization (i) is feasible even for low-level systems code, such as network stacks, (ii) improves system performance without the need for complex cost models, (iii) incurs low developer effort, especially compared to manual exploration. We conclude with future opportunities online system code specialization enables.

Related papers

• MAS-PromptBench: When Does Prompt Optimization Improve Multi-Agent LLM Systems? [12.31270556465256]
  Multi-agent systems (MAS) offer a scalable path forward for agentic AI.<n>System prompts specify agents' roles and behaviors, enabling system-level improvements without model finetuning.<n>It remains unclear whether, when, and by how much prompt optimization improves MAS performance.
  arXiv  Detail & Related papers  (2026-06-22T17:48:40Z)
• The Time is Here for Just-in-Time Systems: Challenges and Opportunities [102.80383421480155]
  Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments.<n>We argue that LLM-based coding agents now make a different approach tractable: Just-in-Time Systems.<n>We present a JIT system synthesis pipeline, Jitskit, and explore its effectiveness in synthesizing key-value stores from spec cards.
  arXiv  Detail & Related papers  (2026-05-22T18:03:41Z)
• AgenticRecTune: Multi-Agent with Self-Evolving Skillhub for Recommendation System Optimization [13.524716959938237]
  We propose AgenticRecTune to manage the end-to-end configuration optimization workflow.<n>The framework comprises five specialized agents, Actor, Critic, Insight, Skill, and Online.<n>We also introduce a self-evolving Skillhub, which utilizes a collaboration between the Insight Agent and Skill Agent to summarize the history results.
  arXiv  Detail & Related papers  (2026-04-21T23:15:58Z)
• An Empirical Study on the Effects of System Prompts in Instruction-Tuned Models for Code Generation [4.76360912129794]
  We systematically evaluate how system prompts affect code assistant.<n>We find that increasing system-prompt constraint specificity does not monotonically improve correctness.<n>For larger code-specialized models, few-shot examples can degrade performance relative to zero-shot generation.
  arXiv  Detail & Related papers  (2026-02-16T22:11:21Z)
• Not All Tokens Matter: Data-Centric Optimization for Efficient Code Summarization [46.365359894614706]
  We evaluate how system prompts affect ILMs and CLMs in code generation tasks.<n>Our evaluation framework, spanning 120 model configurations, reveals that the influence of system prompts increases with model scale.<n>Java shows greater sensitivity to system prompt variations than Python.
  arXiv  Detail & Related papers  (2026-01-28T00:45:28Z)
• Mixture-of-Schedulers: An Adaptive Scheduling Agent as a Learned Router for Expert Policies [45.88210980548133]
  We present the Adaptive Scheduling Agent (ASA), a lightweight framework that intelligently matches workloads to the most suitable scheduling policy at runtime.<n>ASA's core is a novel, low-overhead offline/online approach.<n>Our evaluation, based on a novel benchmark focused on user-experience metrics, demonstrates that ASA consistently outperforms the default Linux scheduler.
  arXiv  Detail & Related papers  (2025-11-07T14:16:31Z)
• Prompting for Performance: Exploring LLMs for Configuring Software [10.764205218438759]
  Large language models (LLMs) can assist in performance-oriented software configuration through prompts.<n>We evaluate several LLMs on tasks including identifying relevant options, ranking configurations, and recommending performant configurations across various systems.
  arXiv  Detail & Related papers  (2025-07-13T21:05:01Z)
• Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewards [95.19837878559456]
  We propose Optimas, a unified framework for effective optimization of compound systems.<n>In each iteration, Optimas efficiently adapts the Local Reward Function (LRF) to maintain this property while simultaneously maximizing each component's local reward.<n>We present extensive evaluations across five real-world compound systems to demonstrate that Optimas outperforms strong baselines by an average improvement of 11.92%.
  arXiv  Detail & Related papers  (2025-07-03T07:12:48Z)
• SysLLMatic: Large Language Models are Software System Optimizers [2.4416377721219145]
  We present SysLLMatic, a system that integrates Large Language Models with profiling-guided feedback and system performance insights.<n>We evaluate it on three benchmark suites: HumanEval_Bench (competitive programming in C++), SciMark2 (scientific kernels in Java), and DaCapoBench (large-scale software systems in Java)
  arXiv  Detail & Related papers  (2025-06-02T01:57:21Z)
• System Prompt Optimization with Meta-Learning [60.04718679054704]
  We introduce the novel problem of bilevel system prompt optimization, whose objective is to design system prompts that are robust to diverse user prompts.<n>We propose a meta-learning framework, which meta-learns the system prompt by optimizing it over various user prompts across multiple datasets.<n>We conduct experiments on 14 unseen datasets spanning 5 different domains, on which we show that our approach produces system prompts that generalize effectively to diverse user prompts.
  arXiv  Detail & Related papers  (2025-05-14T16:46:15Z)
• KIMAs: A Configurable Knowledge Integrated Multi-Agent System [46.91903900679881]
  This technical report presents a knowledge integrated multi-agent system, KIMAs, to address these challenges. Our work provides a scalable framework for advancing the deployment of large language models in real-world settings.
  arXiv  Detail & Related papers  (2025-02-13T18:51:12Z)
• A quantitative framework for evaluating architectural patterns in ML systems [49.1574468325115]
  This study proposes a framework for quantitative assessment of architectural patterns in ML systems.<n>We focus on scalability and performance metrics for cost-effective CPU-based inference.
  arXiv  Detail & Related papers  (2025-01-20T15:30:09Z)
• LLM-based Optimization of Compound AI Systems: A Survey [64.39860384538338]
  In a compound AI system, components such as an LLM call, a retriever, a code interpreter, or tools are interconnected. Recent advancements enable end-to-end optimization of these parameters using an LLM. This paper presents a survey of the principles and emerging trends in LLM-based optimization of compound AI systems.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• SPRIG: Improving Large Language Model Performance by System Prompt Optimization [45.96513122345295]
  Large Language Models (LLMs) have shown impressive capabilities in many scenarios, but their performance depends on the choice of prompt. We propose SPRIG, an edit-based genetic algorithm that iteratively constructs prompts from prespecified components to maximize the model's performance in general scenarios. We evaluate the performance of system prompts on a collection of 47 different types of tasks to ensure generalizability.
  arXiv  Detail & Related papers  (2024-10-18T18:51:44Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• LAMBO: Large AI Model Empowered Edge Intelligence [71.56135386994119]
  Next-generation edge intelligence is anticipated to benefit various applications via offloading techniques. Traditional offloading architectures face several issues, including heterogeneous constraints, partial perception, uncertain generalization, and lack of tractability. We propose a Large AI Model-Based Offloading (LAMBO) framework with over one billion parameters for solving these problems.
  arXiv  Detail & Related papers  (2023-08-29T07:25:42Z)
• BAGUA: Scaling up Distributed Learning with System Relaxations [31.500494636704598]
  BAGUA is a new communication framework for distributed data-parallel training. Powered by the new system design, BAGUA has a great ability to implement and extend various state-of-the-art distributed learning algorithms. In a production cluster with up to 16 machines, BAGUA can outperform PyTorch-DDP, Horovod and BytePS in the end-to-end training time.
  arXiv  Detail & Related papers  (2021-07-03T21:27:45Z)
• Multi-Exit Semantic Segmentation Networks [78.44441236864057]
  We propose a framework for converting state-of-the-art segmentation models to MESS networks. specially trained CNNs that employ parametrised early exits along their depth to save during inference on easier samples. We co-optimise the number, placement and architecture of the attached segmentation heads, along with the exit policy, to adapt to the device capabilities and application-specific requirements.
  arXiv  Detail & Related papers  (2021-06-07T11:37:03Z)
• RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance [6.489720534548981]
  RecPipe is a system to jointly optimize recommendation quality and inference performance. RPAccel is a custom accelerator that jointly optimize quality, tail-latency, and system throughput.
  arXiv  Detail & Related papers  (2021-05-18T20:44:04Z)
• MLComp: A Methodology for Machine Learning-based Performance Estimation and Adaptive Selection of Pareto-Optimal Compiler Optimization Sequences [10.200899224740871]
  We propose a novel Reinforcement Learning-based policy methodology for embedded software optimization. We show that different Machine Learning models are automatically tested to choose the best-fitting one. We also show that our framework can be trained efficiently for any target platform and application domain.
  arXiv  Detail & Related papers  (2020-12-09T19:13:39Z)
• Centralized and distributed online learning for sparse time-varying optimization [2.7006549420999937]
  Development of online algorithms to track time-varying systems has drawn a lot of attention in the last years. We propose an online learning approach, which does not employ a given model and is suitable for adversarial frameworks. Specifically, we develop centralized and distributed algorithms, and we theoretically analyze them in terms of dynamic regret.
  arXiv  Detail & Related papers  (2020-01-31T16:29:38Z)
• Optimal by Design: Model-Driven Synthesis of Adaptation Strategies for Autonomous Systems [9.099295007630484]
  We present Optimal by Design (ObD), a framework for model-based requirements-driven synthesis of optimal adaptation strategies for autonomous systems. ObD proposes a model for the high-level description of the basic elements of self-adaptive systems, namely the system, capabilities, requirements and environment. Based on those elements, a Markov Decision Process (MDP) is constructed to compute the optimal strategy or the most rewarding system behaviour.
  arXiv  Detail & Related papers  (2020-01-16T12:49:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.