Minimal-Intervention KV Retention via Set-Conditioned Diversity

• URL: http://arxiv.org/abs/2605.14292v2
• Date: Sun, 17 May 2026 02:21:29 GMT
• Title: Minimal-Intervention KV Retention via Set-Conditioned Diversity
• Abstract summary: KV-cache compression at small budgets is a crowded design space spanning cache representation, head-wise routing, compression cadence, decoding behavior, and within-budget scoring.<n>We propose a one-function modification to the TriAttentioncitemao2026triattention retention scorer that replaces argmax-top-$k$ with greedy facility-location-inspired selection.<n>A pre-registered protocol tunes $$ on a frozen development split and confirms on a disjoint held-out split; with $= 0.5$, $$ clears Bonferroni on two of the four (
• Score: 7.208745673318648
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: KV-cache compression at small budgets is a crowded design space spanning cache representation, head-wise routing, compression cadence, decoding behavior, and within-budget scoring. We study seven mechanisms across these five families under matched mean cache on long-form mathematical reasoning (MATH-500~\cite{hendrycks2021math}) with two distilled-reasoning models (Qwen-7B and Llama-8B variants of DeepSeek-R1-Distill~\cite{deepseek2025r1}) at budgets $b \in \{64, 128\}$. All seven were rejected. We then propose $α$, a one-function modification to the TriAttention~\cite{mao2026triattention} retention scorer that replaces argmax-top-$k$ with greedy facility-location-inspired selection under a V-space redundancy penalty controlled by a single weight $λ$. A pre-registered protocol tunes $λ$ on a frozen development split and confirms on a disjoint held-out split; with $λ= 0.5$, $α$ clears Bonferroni on two of the four (model, budget) cells (Qwen $b{=}128$ and Llama $b{=}64$), no cell is significantly negative, and the pre-registered Branch~A triggers. The finding is asymmetric: a minimal scoring modification beat heavier structural redesigns in this regime, and the combined matched-memory, sympy-graded, held-out confirmation protocol is the evidence standard that made the asymmetry visible.

Related papers

• Fixed-order postselected CHSH reference acquisition for parity-constrained spatial-mode qubits on a commercial cloud photonic processor [0.0]
  We report a fixed-order Clauser-Horne-Shimony-Holt (CHSH) acquisition and reporting protocol for two encoded photonic qubits on Quandela's commercial, cloud-accessible Belenos processor.<n>Each logical qubit is a two-dimensional spatial-mode subspace in the seven-dimensional zero-sum sector of an eight-mode single-photon register.
  arXiv  Detail & Related papers  (2026-08-13T01:08:02Z)
• Approximate Speculative Decoding [2.7677713446315946]
  Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel.<n>We introduce textbf Speculative Decoding (ASD), a training-free verifier that replaces binary first-mismatch truncation with budgeted longest selection.<n>ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero.
  arXiv  Detail & Related papers  (2026-08-04T10:45:24Z)
• Top-$k$ Pareto Bandits: Hypervolume Regret for Multi-Objective Slate Selection [48.83076933238825]
  We consider a multi-objective bandit problem where, at each round, the agent selects a slate of $k$ arms and observes their $d$-dimensional reward vectors under semi-bandit feedback.<n>We formalize this objective through the dominated hypervolume induced by the selected subset of arms, and define an $$-approximate hypervolume regret with respect to the best size-$k$ subset achievable in hindsight.<n>We establish a gap-free regret bound $tildeO(dsqrtnkT)$ that holds on every instance, together with a gap
  arXiv  Detail & Related papers  (2026-07-28T21:10:39Z)
• Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders [9.291632832739822]
  We introduce two sparsity regularizers compatible with the Top-$k$ architecture.<n>Our central finding is that hard architectural sparsity and soft sparsity regularization are complementary rather than mutually exclusive.
  arXiv  Detail & Related papers  (2026-06-25T17:34:39Z)
• When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation [9.055086193088083]
  We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
  arXiv  Detail & Related papers  (2026-05-25T15:57:11Z)
• Hidden-State Privacy Has an Empty Middle [51.56484100374058]
  We prove a complementary Fisher-ball lower bound: every full-rank Gaussian release at $O(1)$ Fisher utility admits a direction whose Mahalanobis signal grows linearly in hidden width.<n>A split-memory transformer trained from scratch reaches $G_mathrmMah in [20, 33]$ at 90M and maintains a $6$--$24times$ advantage over same-budget GPT baselines from 30M to 1B at a fixed-token language loss penalty.
  arXiv  Detail & Related papers  (2026-05-21T20:12:09Z)
• Sequential Minimal Optimization for $\varepsilon$-SVR with MAPE Loss and Sample-Dependent Box Constraints [0.0]
  We derive a Sequential Minimal Optimization (SMO) algorithm for the quadratic dual problem arising from $varepsilon$-SVRciteVapnik1995, Drucker1997, Smola2004 modified to minimize the Mean Absolute Percentage Error (MAPE)citeMakridakis1993, Hyndman2006.<n>An implementation is available in the open-source textttpsvr R packageciteBenavidesHerrera2026Rpsvr.
  arXiv  Detail & Related papers  (2026-05-02T13:51:46Z)
• ZenBrain: A Neuroscience-Inspired 7-Layer Memory Architecture for Autonomous AI Systems [51.56484100374058]
  On LongMemEval-500, ZenBrain matches a long-context oracle's binary-judge accuracy to within 4.5 pp.<n> ZenBrain is a 7-layer neuroscience-inspired memory architecture.
  arXiv  Detail & Related papers  (2026-04-26T20:39:19Z)
• Learning Hippo: Multi-attractor Dynamics and Stability Effects in a Biologically Detailed CA3 Extension of Hopfield Networks [0.0]
  We present a biologically detailed extension of the classical Hopfield/Marr auto-associative memory model for CA3.<n>We implement ten populations (two asymmetric pyramidal subtypes, eight GABAergic interneuron classes), forty-seven compartments, multi-rule plasticity, and a bimodal cholinergic encoding/consolidation cycle.
  arXiv  Detail & Related papers  (2026-04-22T15:28:17Z)
• Training Transformers in Cosine Coefficient Space [0.0]
  A transformer trained from scratch on character-level Shakespeare reaches validation loss $1.604$ at $K = mn/2$.<n>A rank-48 LoRA factorization at the same trainable parameter count reaches only $1.801$.<n>A random orthonormal basis matches the DCT within noise at $K = mn/2$, and a compression sweep through $K = mn/10$ and $K = mn/20$ shows that subspaces that can host high-rank matrices keep the loss low.
  arXiv  Detail & Related papers  (2026-04-06T05:39:31Z)
• Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection [15.230462656535034]
  Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = dmodel$)<n>Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary.<n>We argue that selection is an inherently lower-dimensional operation than value transfer, requiring only $BigO(log N) dimensions to distinguish among $N relevant patterns.
  arXiv  Detail & Related papers  (2026-02-16T23:45:39Z)
• Burau representation, Squier's form, and non-Abelian anyons [53.92822954974537]
  We introduce a frequency-tunable, two-dimensional non-Abelian control of operation order constructed from the Burau representation of the braid group $B_3$.<n>The sign change of $Delta(omega)$ across the Squier positivity window reveals alternating constructive and destructive interference of causal orders.<n> Numerical simulations confirm both enhancement and suppression regimes, establishing a minimal $B_3$ braid control.
  arXiv  Detail & Related papers  (2025-10-21T00:25:21Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Fine-Grained Gap-Dependent Bounds for Tabular MDPs via Adaptive Multi-Step Bootstrap [84.66885506098724]
  This paper presents a new model-free algorithm for episodic finite-horizon Markov Decision Processes (MDP), Adaptive Multi-step Bootstrap (AMB) We show AMB achieves a gap-dependent regret bound that only scales with the sum of the inverse of the sub-optimality gaps. We also show AMB suffers an additional $frac|Z_mul|Delta_min$ regret, where $Z_mul$ is the set of state-action pairs $(s,a)$'s satisfying $a$ is a non-unique optimal action for
  arXiv  Detail & Related papers  (2021-02-09T07:46:34Z)
• Towards Defending Multiple $\ell_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization [120.99395850108422]
  Existing adversarial defenses typically improve model robustness against individual specific perturbations. Some recent methods improve model robustness against adversarial attacks in multiple $ell_p$ balls, but their performance against each perturbation type is still far from satisfactory. We propose Gated Batch Normalization (GBN) to adversarially train a perturbation-invariant predictor for defending multiple $ell_p bounded adversarial perturbations.
  arXiv  Detail & Related papers  (2020-12-03T02:26:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.