Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit

• URL: http://arxiv.org/abs/2605.28873v1
• Date: Mon, 25 May 2026 07:13:35 GMT
• Title: Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit
• Abstract summary: We adapt the classical paired sample-size calculation to quantization benchmarks.<n>The bound turns "how reliable is my quantization claim?" into a one-line budget.<n>We complement the bound with a five-line pre-registration template.
• Score: 0.30586855806896046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This is a planning-method note with an unpaired pilot audit. We adapt the classical paired-binary sample-size calculation (Miettinen, 1968) to quantization benchmarks, giving a conservative minimum detectable effect (MDE) bound $δ^{*} \le (z_{1-α/2}+z_{1-β})\sqrt{ρ_d/m}$ in the paired item count $m$ and the FP16-NF4 disagreement rate $ρ_d$. The bound turns "how reliable is my quantization claim?" into a one-line budget a benchmark designer can commit to before running. We illustrate the bound on four models and four benchmarks ($k=5$ splits of $n=100$), and add a parallel MMLU prompt-template study to put the bound's quantization-noise scale alongside the prompt-noise scale. Assuming $ρ_d=0.10$ (an unmeasured planning value), all observed NF4-FP16 deltas fall below the implied MDE, and most cross-split SDs lie within $\pm 1.5$ pp of the binomial reference $\sqrt{p(1-p)/n}$, so much of the variance reported as "benchmark unreliability" on $n=100$ subsamples is binomial sampling noise. The single borderline cell (OPT-WinoGrande, $|Δ|=3.2$ pp) is below the implied MDE at $ρ_d=0.10$ but above it at $ρ_d=0.05$, illustrating the planning trade-off the bound makes explicit. On MMLU, prompt-template ranges of 2-10 pp meet or exceed the largest observed quantization delta (3.2 pp), so a quantization audit that does not first fix the prompt template absorbs template variance into its noise floor. We complement the bound with a five-line pre-registration template.

Related papers

• Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances [5.053749663617978]
  Quantized Mixture-of-Experts (MoE) services can hold several pre-materialized instances of one base model.<n>Because instance materialization and replica counts consume memory, we treat them as upstream provisioning decisions.<n>We route each request to maximize modeled throughput under a class-level expected quality-degradation budget.
  arXiv  Detail & Related papers  (2026-09-11T07:57:25Z)
• SSTQ:Privacy-Preserving Vector Quantization via Subsampled Stochastic TurboQuant [79.24089819400126]
  Subsampled TurboQuant (SSTQ) is a framework that combines overcomplete equal-width tight frames, coordinate subsampling, and privacy-aware quantization.<n>SSTQ achieves mean squared error scaling while using only $lceil log N il + b$ bits per client, where $N = (d)$ is the frame size.<n>We also derive a surrogate privacy-aware codebook objective that reduces the codebook-dependent MSE scaling from $O(4b)$ to $O(2b)$.
  arXiv  Detail & Related papers  (2026-08-05T17:51:25Z)
• ForgettingOT: Certified Speculative Batching from Sinkhorn's Projective Forgetting [0.0]
  We show that the eigenmode of the fixed-point Jacobian $J_tstar=QP=Pstar P$ controls the strict correction tail.<n>ForgettingOT turns this nonlinear Perron--Frobenius fact into a certified executor for streams of related Sinkhorn problems.
  arXiv  Detail & Related papers  (2026-07-27T17:59:36Z)
• An End-to-End Hybrid Quantum--Classical Sampling Workflow for Discrete Markov Random Fields: A Reproducible Case Study [0.0]
  We study amplitude-encoded i.i.d. sampling for small discrete Markov random fields (MRFs)<n>This removes quantum exponential speedup but allows a clean comparison against classical MCMC based on independent circuit samples.<n>A matched-budget VQC vs. MPS comparison shows VQC fidelities fall far below MPS: $(F_mathrmVQC, F_mathrmMPS) = (0.31, 0.96), (0.21, 0.96), (0.17, 0.88)$ at compressions $10.7times$, $34.1times$, and $11
  arXiv  Detail & Related papers  (2026-07-10T18:33:14Z)
• Trade-off Functions for DP-SGD with Subsampling based on Random Shuffling: Tight Upper and Lower Bounds [7.787109481104569]
  We derive a tight analysis of the trade-off function for Differentially Private Gradient Descent (DP-SGD) with subsampling based on random shuffling.<n>Our concrete bounds, derived via the Berry-Esseen theorem, are tight up to constant factors within the proof framework.
  arXiv  Detail & Related papers  (2026-05-07T13:35:43Z)
• SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection [0.0]
  SpecKV is a lightweight adaptive controller that selects $$ per speculation step using signals extracted from the draft model itself.<n>We demonstrate that the optimal $$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate.
  arXiv  Detail & Related papers  (2026-05-04T17:55:05Z)
• Posterior Sampling by Combining Diffusion Models with Annealed Langevin Dynamics [6.987640034932562]
  Posterior sampling provides an accurate and fair framework for tasks such as inpainting, deblurring, and MRI reconstruction.<n>We prove that combining diffusion models with an variant of Langevin dynamics achieves conditional sampling in time using merely an $L4$ bound on the score error.
  arXiv  Detail & Related papers  (2025-10-30T10:17:27Z)
• Entropic Risk Optimization in Discounted MDPs: Sample Complexity Bounds with a Generative Model [2.4145441422386464]
  We analyze the sample complexities of learning the optimal state-action value function $Q*$ and an optimal policy $pi*$ in a finite discounted decision process.<n>We provide and analyze a simple model based approach which we call model-based risk-sensitive $Q$-value-iteration (MB-RS-QVI)
  arXiv  Detail & Related papers  (2025-05-30T22:27:57Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• Fast Rates for Bandit PAC Multiclass Classification [73.17969992976501]
  We study multiclass PAC learning with bandit feedback, where inputs are classified into one of $K$ possible labels and feedback is limited to whether or not the predicted labels are correct. Our main contribution is in designing a novel learning algorithm for the agnostic $(varepsilon,delta)$PAC version of the problem.
  arXiv  Detail & Related papers  (2024-06-18T08:54:04Z)
• Near Sample-Optimal Reduction-based Policy Learning for Average Reward MDP [58.13930707612128]
  This work considers the sample complexity of obtaining an $varepsilon$-optimal policy in an average reward Markov Decision Process (AMDP) We prove an upper bound of $widetilde O(H varepsilon-3 ln frac1delta)$ samples per state-action pair, where $H := sp(h*)$ is the span of bias of any optimal policy, $varepsilon$ is the accuracy and $delta$ is the failure probability.
  arXiv  Detail & Related papers  (2022-12-01T15:57:58Z)
• Reward-Mixing MDPs with a Few Latent Contexts are Learnable [75.17357040707347]
  We consider episodic reinforcement learning in reward-mixing Markov decision processes (RMMDPs) Our goal is to learn a near-optimal policy that nearly maximizes the $H$ time-step cumulative rewards in such a model.
  arXiv  Detail & Related papers  (2022-10-05T22:52:00Z)
• Best Policy Identification in Linear MDPs [70.57916977441262]
  We investigate the problem of best identification in discounted linear Markov+Delta Decision in the fixed confidence setting under a generative model. The lower bound as the solution of an intricate non- optimization program can be used as the starting point to devise such algorithms.
  arXiv  Detail & Related papers  (2022-08-11T04:12:50Z)
• The Sample Complexity of Robust Covariance Testing [56.98280399449707]
  We are given i.i.d. samples from a distribution of the form $Z = (1-epsilon) X + epsilon B$, where $X$ is a zero-mean and unknown covariance Gaussian $mathcalN(0, Sigma)$. In the absence of contamination, prior work gave a simple tester for this hypothesis testing task that uses $O(d)$ samples. We prove a sample complexity lower bound of $Omega(d2)$ for $epsilon$ an arbitrarily small constant and $gamma
  arXiv  Detail & Related papers  (2020-12-31T18:24:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.