The Time is Here for Just-in-Time Systems: Challenges and Opportunities

• URL: http://arxiv.org/abs/2605.24096v1
• Date: Fri, 22 May 2026 18:03:41 GMT
• Title: The Time is Here for Just-in-Time Systems: Challenges and Opportunities
• Abstract summary: Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments.<n>We argue that LLM-based coding agents now make a different approach tractable: Just-in-Time Systems.<n>We present a JIT system synthesis pipeline, Jitskit, and explore its effectiveness in synthesizing key-value stores from spec cards.
• Score: 102.80383421480155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Core systems like key-value stores have historically taken years to build, and are designed to be general so as to amortize cost across deployments, paying a significant performance cost. We argue that LLM-based coding agents now make a different approach tractable: Just-in-Time Systems, in which the entire system is synthesized from scratch, specialized to the environment, workload, and required system properties. We present a JIT system synthesis pipeline, Jitskit, and explore its effectiveness in synthesizing key-value stores from spec cards that span different YCSB workloads, deployment constraints (e.g., compute resources), and system properties (e.g., consistency and durability). Jitskit iteratively refines a system implementation to match the specification against an evolving evaluation test suite. The resulting synthesized systems are performant, beating comparable state-of-the-art systems on 18 of 18 specs tried, by up to 4.6x over the best off-the-shelf baseline on the most favorable spec. Naively running Claude Code either reward-hacks or underperforms Jitskit by up to 5.4x. We discuss the challenges we overcame in building Jitskit and our key takeaways.

Related papers

• LATTEArena: An Evaluation Framework for LLM-powered Tabular Feature Engineering (Extended Version) [21.873402103599588]
  We introduce LATTEArena, a standardized, modular, and benchmarking framework for automated feature engineering.<n>By enabling controlled component-level comparisons, LATTEArena shifts the paradigm from ad-hoc prompt engineering to systematic context management.<n>All code, datasets, and over 4,000 execution logs are publicly available to foster a dynamic, community-driven benchmark.
  arXiv  Detail & Related papers  (2026-06-08T04:03:54Z)
• Inductive Deductive Synthesis: Enabling AI to Generate Formally Verified Systems [100.24694338574402]
  We present Inductive Deductive Synthesis (IDS), which jointly and incrementally synthesizes implementation and proof.<n>IDS achieves 7/7 in about 6.8 hours and $106 per spec on average, roughly 200x faster than expert effort and 17% cheaper than SOTA agents.<n>IDS further incorporates performance feedback into the same loop, yielding implementations up to 3x faster than published verified systems.
  arXiv  Detail & Related papers  (2026-05-22T00:05:36Z)
• VibeServe: Can AI Agents Build Bespoke LLM Serving Systems? [7.395579663607589]
  We propose VibeServe, the first agentic loop that generates entire LLM serving stacks end-to-end.<n>In the standard deployment setting, where existing stacks are highly optimized, VibeServe remains competitive with vLLM.<n>In non-standard scenarios, VibeServe outperforms existing systems by exploiting opportunities that generic systems miss.
  arXiv  Detail & Related papers  (2026-05-07T11:54:53Z)
• CocoaBench: Evaluating Unified Digital Agents in the Wild [98.84044600506597]
  We introduce CocoaBench, a benchmark for unified digital agents built from human-designed, long-horizon tasks.<n>We also present CocoaAgent, a lightweight shared scaffold for controlled comparison across model backbones.
  arXiv  Detail & Related papers  (2026-04-13T09:00:10Z)
• MASEval: Extending Multi-Agent Evaluation from Models to Systems [77.09381093359598]
  MASEval is a framework-agnostic library that treats the entire system as the unit of analysis.<n>It allows researchers to explore all components of agentic systems, opening new avenues for principled system design.
  arXiv  Detail & Related papers  (2026-03-09T18:46:17Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• SysMoBench: Evaluating AI on Formally Modeling Complex Real-World Systems [12.181911851729614]
  We present SysMoBench, a benchmark that evaluates AI's ability to formally model large, complex systems.<n>We focus on concurrent and distributed systems, which are keystones of today's critical computing infrastructures.
  arXiv  Detail & Related papers  (2025-09-27T05:24:54Z)
• DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis [52.636738269442766]
  We introduce DeepScholar-bench, a live benchmark and holistic, automated evaluation framework designed to evaluate generative research synthesis systems.<n>DeepScholar-bench draws queries from recent, high-quality ArXiv papers and focuses on a real research synthesis task.<n>We also develop DeepScholar-base, a reference pipeline implemented efficiently using the LOTUS API.
  arXiv  Detail & Related papers  (2025-08-27T16:36:34Z)
• tieval: An Evaluation Framework for Temporal Information Extraction Systems [2.3035364984111495]
  Temporal information extraction has attracted a great deal of interest over the last two decades. Having access to a large volume of corpora makes it difficult when it comes to benchmark TIE systems. tieval is a Python library that provides a concise interface for importing different corpora and facilitates system evaluation.
  arXiv  Detail & Related papers  (2023-01-11T18:55:22Z)
• BAGUA: Scaling up Distributed Learning with System Relaxations [31.500494636704598]
  BAGUA is a new communication framework for distributed data-parallel training. Powered by the new system design, BAGUA has a great ability to implement and extend various state-of-the-art distributed learning algorithms. In a production cluster with up to 16 machines, BAGUA can outperform PyTorch-DDP, Horovod and BytePS in the end-to-end training time.
  arXiv  Detail & Related papers  (2021-07-03T21:27:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.