Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start

• URL: http://arxiv.org/abs/2604.06664v1
• Date: Wed, 08 Apr 2026 04:31:34 GMT
• Title: Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start
• Abstract summary: Cold-start latency remains a major bottleneck for modern LLM service providers.<n>Existing approaches either rely on brittle kernel-specific patching or executable process context.<n>We present a Foundry template-based graph materialization system that persists both graph topology and execution context.
• Score: 33.16822047715293
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Modern LLM service providers increasingly rely on autoscaling and parallelism reconfiguration to respond to rapidly changing workloads, but cold-start latency remains a major bottleneck. While recent systems have reduced model weight loading to seconds, CUDA graph capture still takes tens of seconds to minutes and often dominates startup. Unfortunately, CUDA graphs cannot be naively serialized: beyond graph topology, they are tightly coupled to execution context, including device addresses embedded in kernel arguments and kernel code lazily loaded during warmup. Existing approaches either rely on brittle kernel-specific patching or heavyweight process-level checkpoint/restore that are inflexible to dynamic parallelism switching. We present Foundry, a template-based CUDA graph context materialization system that persists both graph topology and execution context during an offline processing stage, and reconstructs executable graphs online with negligible overhead. Foundry enforces deterministic memory layouts, automatically extracts and reloads kernel binaries required by captured graphs, and reduces online reconstruction costs through topology-based templating. For distributed serving, Foundry further enables a single-GPU offline capture to generate templates for multi-GPU deployments by patching only rank-dependent communication state. Across dense and MoE models up to 235B parameters, Foundry reduces cold-start latency by up to 99%, cutting the initialization time of Qwen3-235B-A22B from 10 minutes to 3.9 seconds while preserving the throughput gains of CUDA graphs.

Related papers

• Accelerating Dynamic Graph Clustering on GPU Architectures with cuGraph [0.9581150516892287]
  This work addresses community detection in temporal networks through GPU-accelerated extensions of spectral clustering and modularity-based algorithms originally designed for static graphs.<n>We release our implementation as free and open-source software, including Python bindings through the NetworkX-Temporal library for ease of use and zero-code acceleration with existings.
  arXiv  Detail & Related papers  (2026-08-04T14:01:00Z)
• Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs [27.795876706914857]
  This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs)<n>We propose Gleam, a novel and network-efficient framework for task-generic GPU sharing.<n>Gleem consistently outperforms state-of-the-art baselines, achieving 1.4-24.2 times improvements in API remoting efficiency.
  arXiv  Detail & Related papers  (2026-07-25T09:06:47Z)
• Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving [0.5076419064097734]
  We study the opposite regime: low-latency, small-batch, on-device physical-AI serving.<n>We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state.<n>This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries.
  arXiv  Detail & Related papers  (2026-06-18T17:49:36Z)
• NOMAD: Generating Embeddings for Massive Distributed Graphs [10.29444459089447]
  Successful machine learning on graphs or networks requires embeddings that represent nodes and edges as low-dimensional vectors.<n>We present NOMAD, a distributed-memory graph embedding framework using the Message Passing Interface (MPI) for distributed graphs.
  arXiv  Detail & Related papers  (2026-04-10T15:30:43Z)
• S2GS: Streaming Semantic Gaussian Splatting for Online Scene Understanding and Reconstruction [57.07346645250984]
  Streaming Semantic Gaussian Splatting (S2GS) is a strictly causal, incremental 3D Gaussian semantic field framework.<n>It does not leverage future frames and continuously updates scene geometry, appearance, and instance-level semantics without reprocessing historical frames.<n>S2GS matches or outperforms strong offline baselines on joint reconstruction-and-understanding benchmarks.
  arXiv  Detail & Related papers  (2026-03-15T05:48:55Z)
• Nexus:Proactive Intra-GPU Disaggregation of Prefill and Decode in LLM Serving [4.309392302169281]
  Engine-level prefill-decode (PD) disaggregation avoids interference but incurs higher hardware and coordination overhead.<n>PD achieves up to 2.2x higher throughput, 20x lower TTFT, and 2.5x lower TBT than vLLM; outperforms SG by up to 2x; and matches or exceeds disaggregated vLLM.
  arXiv  Detail & Related papers  (2025-07-09T07:27:18Z)
• DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation [61.59996525424585]
  DIFFVSGG is an online VSGG solution that frames this task as an iterative scene graph update problem.<n>We unify the decoding of object classification, bounding box regression, and graph generation three tasks using one shared feature embedding.<n>DIFFVSGG further facilitates continuous temporal reasoning, where predictions for subsequent frames leverage results of past frames as the conditional inputs of LDMs.
  arXiv  Detail & Related papers  (2025-03-18T06:49:51Z)
• SPEED: Streaming Partition and Parallel Acceleration for Temporal Interaction Graph Embedding [22.68416593780539]
  We introduce a novel training approach namely Streaming Edge Partitioning and Parallel Acceleration for Temporal Interaction Graph Embedding. Our method can achieve a good balance in computing resources, computing time, and downstream task performance. Empirical validation across 7 real-world datasets demonstrates the potential to expedite training speeds by a factor of up to 19.29x.
  arXiv  Detail & Related papers  (2023-08-27T15:11:44Z)
• Communication-Free Distributed GNN Training with Vertex Cut [63.22674903170953]
  CoFree-GNN is a novel distributed GNN training framework that significantly speeds up the training process by implementing communication-free training. We demonstrate that CoFree-GNN speeds up the GNN training process by up to 10 times over the existing state-of-the-art GNN training approaches.
  arXiv  Detail & Related papers  (2023-08-06T21:04:58Z)
• Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Caching [59.8522166385372]
  Training and inference with graph neural networks (GNNs) on massive graphs has been actively studied since the inception of GNNs. This paper is concerned with minibatch training and inference with GNNs that employ node-wise sampling in distributed settings. We present SALIENT++, which extends the prior state-of-the-art SALIENT system to work with partitioned feature data.
  arXiv  Detail & Related papers  (2023-05-04T21:04:01Z)
• Dynamic Graph Message Passing Networks for Visual Recognition [112.49513303433606]
  Modelling long-range dependencies is critical for scene understanding tasks in computer vision. A fully-connected graph is beneficial for such modelling, but its computational overhead is prohibitive. We propose a dynamic graph message passing network, that significantly reduces the computational complexity.
  arXiv  Detail & Related papers  (2022-09-20T14:41:37Z)
• DOTIN: Dropping Task-Irrelevant Nodes for GNNs [119.17997089267124]
  Recent graph learning approaches have introduced the pooling strategy to reduce the size of graphs for learning. We design a new approach called DOTIN (underlineDrunderlineopping underlineTask-underlineIrrelevant underlineNodes) to reduce the size of graphs. Our method speeds up GAT by about 50% on graph-level tasks including graph classification and graph edit distance.
  arXiv  Detail & Related papers  (2022-04-28T12:00:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.