Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment

• URL: http://arxiv.org/abs/2606.19558v1
• Date: Wed, 17 Jun 2026 19:59:26 GMT
• Title: Displacement Is Not Direction: Evaluating Fidelity Metrics for Quantized LLM Deployment
• Abstract summary: Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality.<n>We find that KLD is strongly correlated with benchmark score over the full cohort.<n>However, this relationship collapses to non-significance in the near-baseline silent zone.
• Score: 0.7249731529275342
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fidelity metrics, such as per-token KL divergence (KLD) against a high-precision reference, are often used in practice as low-cost proxies for benchmark quality. We test this practice on a 28-quant cohort of Qwen3.6-35B-A3B and a 41-quant cohort of Devstral-Small-2-24B, evaluated across a suite of downstream benchmarks. We find that KLD is strongly correlated with benchmark score over the full cohort ($ρ=-0.72$ on Qwen and $ρ=-0.86$ on Devstral, both with $p<0.001$). However, this relationship collapses to non-significance in the near-baseline silent zone ($ρ=+0.00$ on Qwen and $ρ=-0.24$, $p=0.36$, on Devstral). This collapse persists across 14 measurement variants, including different KLD aggregations, perplexity formulations, top-1 agreement, calibration corpora, and context lengths. At the per-prompt level, KLD has only weak failure-prediction power on code, with failed-vs-passed geometric-mean ratios in $[1.08,1.22]$ across five models on LiveCodeBench, and fails as a cross-model router, achieving only $42.3\%-49.4\%$ accuracy on disagreement prompts. We trace the collapse to a structural decomposition: KLD primarily measures the volume of disagreement with the reference, with silent-zone composite $ρ=+0.94$ ($p<0.001$) on Qwen and $+0.55$ ($p=0.03$) on Devstral, while its relationship to the direction of those disagreements is weak and task-conditional.

Related papers

• Online Non-Monotone DR-Submodular Maximization Matching the Offline $0.401$ Factor [47.30196146041476]
  We study asymmetric non-negative, non-monotone DR-submodular functions over compact convex subsets of the $d$-oracle unit.<n>Online algorithm attains factor 0.401$ with sub approximate regret when feedback is conditionally unbiased and bounded.
  arXiv  Detail & Related papers  (2026-09-02T06:00:10Z)
• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence [41.99844472131922]
  A single human must audit $N$ LLM agents under a budget of $B ll N$ audits per round.<n>We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $*$ past which confidence-ranked auditing is emphworse than random.
  arXiv  Detail & Related papers  (2026-07-30T14:52:36Z)
• Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks [10.839956062772792]
  Three agent-safety benchmarks rank the same 18 models differently.<n>R-Judge specificity against AgentHarm safety correlates $-0.64$ at $n=7$ and $+0.02$ at $n=18$, and a quarter of random size-7 subsets reach $|| geq 0.5$ around that near-zero value.<n>AgentHarm shows the strongest held-out association, $=+0.72$ with three-template jailbreak safety after controlling capability.
  arXiv  Detail & Related papers  (2026-07-30T03:45:10Z)
• Which Defense Closes Which Threat? Attributing OWASP-LLM-Top-10 Coverage and Its Brittleness Under Paraphrasing [51.56484100374058]
  Production LLM applications stack several defense families -- refusal-phrase filters, tokenbudget controls, model allowlists, rate limits, tool-registry authentication -- yetBAS benchmarks report a single aggregate coverage number.<n>We add four-LLM-Top-10-aware agents to a 21-agent baseline scanner and target a lattice of four synthetic LLM endpoints.
  arXiv  Detail & Related papers  (2026-06-01T19:39:25Z)
• UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures [0.0]
  We propose emphUR-JEPA, which targets a uniformly $n$-rectifiable measure of local dimension tangent $n$ at small scales.<n>We show that on all four datasets UR--JEPA($mathcalLtextCGLT$) produces a global geometric spectrum with a $4$ to $5$ order-of-magnitude drop at index $sim 20$ to $25$ out of $D = 32$, while LeJEPA's spectrum is near-flat (top-to-bottom ratio at most $3.6$)
  arXiv  Detail & Related papers  (2026-05-31T20:26:54Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• Representation Without Reward: A JEPA Audit for LLM Fine-Tuning [1.2691047660244335]
  Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs.<n>For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head emphand improve the decoded task metric.<n>We test that requirement under a fixed Llama-3.2-1B-Instruct LoRA harness on natural-language-to-regex generation.
  arXiv  Detail & Related papers  (2026-05-14T20:27:32Z)
• Scale-Sensitive Shattering: Learnability and Evaluability at Optimal Scale [54.65053906803857]
  We study the optimal scale at which real-valued function classes exhibit uniform convergence and learnability.<n>Our main result establishes a scale-sensitive generalization of the fundamental theorem of PAC learning.<n>We also highlight several open questions on quantitative sample complexity and evaluability.
  arXiv  Detail & Related papers  (2026-05-13T15:41:30Z)
• Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation [45.56738584872585]
  Neural scaling laws approximate a language model's loss as a power-law function of parameter count $N$ and token count $D$.<n>We show that a collinear design induces an inherent ill-conditioning in the Gauss-Newton least-squares problem.<n>We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation.
  arXiv  Detail & Related papers  (2026-05-08T23:00:17Z)
• A Closed-Form Persistence-Landmark Pipeline for Certified Point-Cloud and Graph Classification [0.0]
  PLACE (Persistence-Landmark Analytic Classification Engine) is a closed-form pipeline for classifying point clouds and graphs.<n>Three quantitative guarantees -- a margin-based excess-risk rate, a closed-form descriptor-selection rule, and a per-prediction certificate -- are derived from training labels alone.
  arXiv  Detail & Related papers  (2026-05-04T17:15:01Z)
• Blind Catalytic Quantum Error Correction: Target-State Estimation and Fidelity Recovery Without \textit{A Priori} Knowledge [0.0]
  We introduce emphblind CQEC, which estimates the target from the noisy output alone before catalytic recovery.<n>Five estimation strategies are benchmarked across three noise models.
  arXiv  Detail & Related papers  (2026-04-13T08:27:08Z)
• Spectral Sentinel: Scalable Byzantine-Robust Decentralized Federated Learning via Sketched Random Matrix Theory on Blockchain [0.0]
  Byzantine clients poison gradients under heterogeneous (Non-IID) data.<n>We propose Spectral Sentinel, a Byzantine detection and aggregation framework.<n>We implement the full system with blockchain integration on Polygon networks.
  arXiv  Detail & Related papers  (2025-12-14T09:43:03Z)
• Convergence Rate Analysis of LION [54.28350823319057]
  LION converges iterations of $cal(sqrtdK-)$ measured by gradient Karush-Kuhn-T (sqrtdK-)$. We show that LION can achieve lower loss and higher performance compared to standard SGD.
  arXiv  Detail & Related papers  (2024-11-12T11:30:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.