Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization

• URL: http://arxiv.org/abs/2607.17019v1
• Date: Sun, 19 Jul 2026 01:13:09 GMT
• Title: Regularize or Localize: When Training-Time KV-Cache Geometry Pays Under Quantization
• Abstract summary: sigreg reduces hidden-state pairwise-cosine anisotropy by $38%$ across three paired seeds.<n>Applying sigreg directly to K and V during continued training, however, reduces mean cache anisotropy by $94%$ across four checkpoints.
• Score: 7.954260884574662
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study whether \sigreg -- LeJEPA's anti-collapse objective -- can reshape representations during standard autoregressive language-model pretraining, and when the resulting geometry helps \kv-cache quantization. We train 110M-parameter models on 10B FineWeb tokens and report three findings. \textbf{(1)} At $λ{=}0.01$, \sigreg reduces hidden-state pairwise-cosine anisotropy by $38\%$ across three paired seeds. Perplexity increases by less than $0.35\%$ in every pair, with no consistent zero-shot loss. \textbf{(2)} This change does not propagate from hidden states to the \kv cache. Applying \sigreg directly to K and V during continued training, however, reduces mean cache anisotropy by $94\%$ across four checkpoints. A matched continuation without the \kv term leaves cache geometry nearly unchanged, and the frozen-trunk retrofits we tested do not reproduce the effect. \textbf{(3)} Under untransformed symmetric group-free quantization, direct \kv regularization is the only training condition that prefers per-channel scaling in all three seeds, and under that same 3-bit per-channel scheme the baseline incurs $4.3$--$7.9\times$ the directly regularized model's \dnll. Under the full simulated KIVI-style configuration (mixed arrangement, zero-points, grouped scales), however, all models reach near-parity, including when storage overhead is approximately matched. In this 110M regime, the training intervention helps when quantizer scales are coarse; the advantage vanishes under the tested combination of token-local grouping, mixed \kv scaling, and zero-points. To our knowledge this is the first training-time \emph{distributional} regularization of standard \kv-cache geometry evaluated against post-hoc cache quantization.

Related papers

• Fast Weight Attention for Continual Learning [75.34672054079408]
  Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state.<n>We derive normalized first-order updates for squared-error regression and negative inner-product objectives.<n>We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization.
  arXiv  Detail & Related papers  (2026-08-27T22:55:11Z)
• High-Dimensional Nonparametric Change-Point Detection via Low-Rank Degree-Three Density Projection [2.1991623402625606]
  Distributional changes can be invisible to means and covariances yet appear in skewness, asymmetric interactions, or other third-order structure.<n>We develop a nonparametric change-point method that retains every degree-at-most-three coefficient of a density while avoiding direct density estimation.
  arXiv  Detail & Related papers  (2026-08-16T01:21:55Z)
• A Design Space Study of Density Matrix Parameterizations for Diffusion-Based Quantum State Tomography [0.0]
  We present the first design space study of density matrix parameterizations for diffusion QST.<n>Our calibration of seven parameterizations at 2- and 3-qubit scales, validated by end-to-end training, reveals that emphgeometric conditioning alone does not predict end-to-end performance.
  arXiv  Detail & Related papers  (2026-08-10T14:05:36Z)
• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs [0.0]
  Key-value (KV) cache has become the dominant memory cost of transformer inference.<n>Low-rank methods factor two-dimensional slices of the cache, or per-head matrices or cross-layer feature blocks, and quantization methods lower the bit-width of every entry.<n>Our method, JoLT, applies a partial Tucker decomposition that compresses only the token and feature axes while leaving the head and layer axes intact.<n>A randomized-SVD variant, FlashJoLT, delivers a 5-13x compression-time speedup at 1024-token context and matched quality.
  arXiv  Detail & Related papers  (2026-07-14T09:23:20Z)
• Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization [58.14514930760722]
  We consider a decentralized setup in which the participants collaboratively train and serve a large neural network.<n>In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant.<n>We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup.
  arXiv  Detail & Related papers  (2026-05-22T10:24:57Z)
• Towards Joint Quantization and Token Pruning of Vision-Language Models [53.978753457744055]
  Token pruning and low-bit quantization are complementary for reducing inference costs.<n>We present a collaborative quantization-and-pruning framework that unifies low-bit inference and deterministic visual-token pruning.<n> Experiments on standard VLM benchmarks show improved robustness over stage-wise baselines under the same low-bit regime.
  arXiv  Detail & Related papers  (2026-04-19T08:18:29Z)
• EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models [8.323540970510809]
  We propose EntropyCache, a training-free KV caching method that uses the maximum entropy of newly decoded token distributions as a constant-cost signal for deciding when to recompute.<n> Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct show that EntropyCache achieves $15.2times$-$26.4times$ speedup on standard benchmarks and $22.4times$-$24.1times$ on chain-of-thought benchmarks, with competitive accuracy and decision overhead accounting for only $0.5%$ of inference time.
  arXiv  Detail & Related papers  (2026-03-19T04:46:34Z)
• Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction [6.908972852063454]
  Hybrid architectures combining state-space models with attention have achieved strong efficiency-quality tradeoffs.<n>We present a surprising finding from analyzing GPT-2 models: textbf88% of attention operations retrieve information already predictable from the model's hidden state.<n>We introduce textbfours (textbfConsolidation-based textbfRouting for textbfAdaptive textbfMemory), a biologically inspired memory consolidation mechanism that gradually distills episodic retrievals into parametric semantic memory
  arXiv  Detail & Related papers  (2026-02-12T17:40:15Z)
• Phase-space entropy at acquisition reflects downstream learnability [54.4100065023873]
  We propose an acquisition-level scalar $S_mathcal B$ based on instrument-resolved phase space.<n>We show theoretically that (S_mathcal B) correctly identifies the phase-space coherence of periodic sampling.<n>$|S_mathcal B|$ consistently ranks sampling geometries and predicts downstream reconstruction/recognition difficulty emphwithout training.
  arXiv  Detail & Related papers  (2025-12-22T10:03:51Z)
• Chebyshev Moment Regularization (CMR): Condition-Number Control with Moment Shaping [0.0]
  We introduce textbfChebyshev Moment Regularization (CMR), a simple, architecture-agnostic loss that directly optimize layer spectra.<n>CMR jointly controls spectral edges via a log-condition proxy shapes and the interior via Chebyshev moments.<n>These results support textbfoptimization-driven spectral preconditioning: directly steering models toward well-conditioned regimes for stable, accurate learning.
  arXiv  Detail & Related papers  (2025-10-17T06:54:41Z)
• Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesis [55.561961365113554]
  3D Gaussian Splatting (3DGS) has demonstrated remarkable effectiveness in novel view synthesis (NVS)<n>In this paper, we introduce Self-Ensembling Gaussian Splatting (SE-GS)<n>We achieve self-ensembling by incorporating an uncertainty-aware perturbation strategy during training.<n> Experimental results on the LLFF, Mip-NeRF360, DTU, and MVImgNet datasets demonstrate that our approach enhances NVS quality under few-shot training conditions.
  arXiv  Detail & Related papers  (2024-10-31T18:43:48Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.