Leyline: KV Cache Directives for Agentic Inference

• URL: http://arxiv.org/abs/2606.01065v1
• Date: Sun, 31 May 2026 07:13:15 GMT
• Title: Leyline: KV Cache Directives for Agentic Inference
• Abstract summary: Leyline is a serving-side primitive for agentic editing.<n>A declarative directive 4-tuple separates what to edit from how to preserve position correctness.<n>A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym.
• Score: 0.11099872871193028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern KV cache management assumes the chatbot workload: prompts arrive once and the cache grows append-only, so prefix caching and forward-only eviction are correct by construction. Agentic LLMs break this assumption. Their conversations evolve through policy-driven editing: failed tool calls are retried, stale outputs dropped, trajectories pivoted. Two distinct cache problems result. First, identical content moves to new positions between turns, invalidating exact-prefix caches even though the underlying KV would still be valid; recent work on position-independent caching for MLA addresses this reuse problem. Second, and this paper's focus, a policy may need to direct the serving system to actively remove or replace a span of cached content and continue without re-prefilling everything that came after. No existing primitive offers this. Production agentic harnesses fall back to re-prefill on every edit, paying full prefix-recomputation cost; kernel-level eviction methods make their own decisions and cannot accept policy directives from outside the kernel. We introduce Leyline, a serving-side primitive that closes this gap. A declarative directive 4-tuple separates what to edit from how to preserve position correctness. The policy declares the edit and its mode (in-place splice or prefix-trimmed re-prefill for semantic forgetting); an architecture-agnostic interface routes to a per-architecture kernel that restores attention math via a closed-form RoPE-rotation correction. The splice kernel lifts replay cache-hit by +11.2 pp and cuts latency by up to 241 ms. A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym. The mechanism is open; the policy space it enables is the agenda.

Related papers

• SkillZip Pro: Execution-Aware Dynamic Compression of Progressively Loaded Skills for Self-Evolving Agents [48.36946103539268]
  Method is an evaluation-free compressor for complete, progressively loaded skill bundles.<n>Method compresses emphacross files, removing content from a reference or subskill when the root or a declared environment contract already provides it.<n>It preserves routing, so every required file and directly callable entry remains reachable after rewriting.
  arXiv  Detail & Related papers  (2026-08-31T13:41:16Z)
• Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework [49.16055123488827]
  coding agents powered by large language models can now generate code patches directly from bug reports.<n>No mechanism independently verifies whether it truly resolves the reported problem.<n>We propose RETRACE, a training-free post-generation verification framework.
  arXiv  Detail & Related papers  (2026-08-09T22:59:13Z)
• Resume Means Resume: A Machine-Checked Conformance Contract for Checkpoint, Interrupt, and Resume Semantics in Workflow Persistence Layers [0.0]
  A framework that persists execution state so a run can be interrupted, survive a crash, and continue must decide what a resume means for effects that already happened.<n>Five widely deployed agent workflow frameworks answer differently, none exposes a machine-checkable contract.<n>LangGraph 1.2.9 records a second resume value and never consults it, persists-invalid state silently, and re-executes durably recorded work after a real SIGKILL.
  arXiv  Detail & Related papers  (2026-08-04T15:45:31Z)
• Models Take Notes at Prefill: KV Cache Can Be Editable and Composable [0.3951796994513004]
  Prefix caching reuses prefill only across an exactly shared prefix, so one changed field invalidates the entire downstream cache.<n>Yet overwriting the field's own key/value vectors and reusing the rest leaves the model acting on the old value.<n>The reason, established causally across four model families, is that at prefill the model has already written the field-conditioned conclusion onto downstream notes.<n>A salient erratum amends the notes; and with chain-of-thought, editing the field alone recovers the decision (1.00 at 8B, 1% compute), while without CoT it is ignored.
  arXiv  Detail & Related papers  (2026-06-14T15:31:23Z)
• Scaling Self-Evolving Agents via Parametric Memory [69.96398842169002]
  Existing memory-augmented LLM agents store past experience exclusively in prompt space.<n>We introduce textttTMEM, a self-evolving parametric memory framework.<n>textttTMEM consistently outperforms summary-based and retrieval-based baselines across different model scales.
  arXiv  Detail & Related papers  (2026-06-03T07:18:31Z)
• Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer? [0.0]
  We propose GroundedCache, an evidence-validated cache router that admits a cached answer only when 4 cheap gates simultaneously hold.<n>We build a six-regime workload that stress-tests cache safety rather than only hit rate, and introduce an operator-facing metric, the unsafe-served rate (USR)<n>Across 2 datasets and 12,000 real-LLM generations(Qwen2.5-7B-Instruct on vLLM with Automatic Prefix Caching), GroundedCache drives USR to 0.0% on every HotpotQA regime and to 1.5% on mtRAG document drift.
  arXiv  Detail & Related papers  (2026-05-26T16:50:02Z)
• Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches [15.961259469095571]
  Prefix caching is a key optimization in Large Language Model (LLM) serving.<n>However, its benefit depends critically on the eviction policy as GPU memory is scarce.<n>We show that different token types within a prompt, including system prompts, user queries, tool outputs, model responses, and chain-of-thought reasoning, exhibit up to 756x variation in reuse rates.<n>We present SAECache, a semantic-adaptive prefix cache eviction policy.
  arXiv  Detail & Related papers  (2026-05-12T18:38:24Z)
• When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning [60.714416943748866]
  In single-stream autoregressive interfaces, the same tokens both update the model state and constitute an irreversible public commitment.<n>We introduce Side-by-Side (SxS) Interleaved Reasoning, which makes disclosure timing a controllable decision within standard autoregressive generation.
  arXiv  Detail & Related papers  (2026-05-05T02:59:58Z)
• HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing [12.303016850029683]
  Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment.<n>One line of work installs new facts by directly modifying base weights through locate-then-edit procedures.<n>A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges.<n>We propose HoReN, a codebook-based editor with enhanced routing built on three ideas.
  arXiv  Detail & Related papers  (2026-05-02T15:51:31Z)
• A Parametric Memory Head for Continual Generative Retrieval [52.66674234249913]
  Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries.<n>We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices.<n>We propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head.
  arXiv  Detail & Related papers  (2026-04-25T17:38:51Z)
• Sparse Prefix Caching for Hybrid and Recurrent LLM Serving [0.7284556903703034]
  We formalize sparse prefix caching as checkpoint placement under a distribution over overlap depths.<n>For use cases where requests share a non-trivial prefix, we show that our method consistently improves the frontier traced by standard asymmetry on real-world data.<n>It preserves exact outputs, does not change the recurrent computation itself or require new recurrent update kernels.
  arXiv  Detail & Related papers  (2026-04-17T09:24:58Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• StepCache: Step-Level Reuse with Lightweight Verification and Selective Patching for LLM Serving [0.0]
  We present StepCache, a backend-agnostic step-level reuse layer that segments outputs into ordered steps.<n>StepCache regenerates only failing regions via selective patching.<n>It reduces mean latency from 2.13 s to 0.67 s, median latency from 2.42 s to 0.01 s, and p95 latency from 3.38 s to 3.30 s.
  arXiv  Detail & Related papers  (2026-03-24T17:19:26Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.