Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

• URL: http://arxiv.org/abs/2604.24827v1
• Date: Mon, 27 Apr 2026 15:46:23 GMT
• Title: Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity
• Abstract summary: We introduce textbfIncompressible Knowledge Probes (IKPs), a benchmark of 1,400 factual questions spanning 7 tiers of obscurity.<n>We calibrate a log-linear mapping from IKP accuracy to parameter count on 89 open-weight models (135M-1,600B) spanning 19 vendors.<n>For Mixture-of-Experts models, total parameters predict knowledge far better than active parameters.
• Score: 0.3951796994513004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Closed-source frontier labs do not disclose parameter counts, and the standard alternative -- inference economics -- carries $2\times$+ uncertainty from hardware, batching, and serving-stack assumptions external to the model. We exploit a tighter intrinsic bound: storing $F$ facts requires at least $F/$(bits per parameter) weights, so measuring how much a model \emph{knows} lower-bounds how many parameters it \emph{has}. We introduce \textbf{Incompressible Knowledge Probes (IKPs)}, a benchmark of 1{,}400 factual questions spanning 7 tiers of obscurity, designed to isolate knowledge that cannot be derived by reasoning or compressed by architectural improvements. We calibrate a log-linear mapping from IKP accuracy to parameter count on 89 open-weight models (135M--1,600B) spanning 19 vendors, achieving $R^2 = 0.917$; leave-one-out cross-validation confirms generalization (median fold error $1.59\times$, $68.5\%$ within $2\times$ and $87.6\%$ within $3\times$). For Mixture-of-Experts models, total parameters predict knowledge ($R^2 = 0.79$) far better than active parameters ($R^2 = 0.51$). We evaluate 188 models from 27 vendors and estimate effective knowledge capacity for all major proprietary frontier models; for heavily safety-tuned models the estimates are lower bounds, since refusal policy can hide tens of percentage points of "refused but known" capacity. The widely-reported saturation of reasoning benchmarks does not imply the end of scaling. Procedural capability compresses under the "Densing Law," but across 96 dated open-weight models the IKP time coefficient is $-0.0010$/month (95\% CI $[-0.0031, +0.0008]$) -- indistinguishable from zero, and rejecting the Densing prediction of $+0.0117$/month at $p < 10^{-15}$. Factual capacity continues to scale log-linearly with parameters across generations and across vendors.

Related papers

• Certification cost of quantum models: measurement correlation, not parameter count [0.0]
  Certifying an empirical Fisher matrix to relative Frobenius error costs $(B p2 V/(varepsilon2 G))$ circuit executions.<n>Cost-optimal readout weights cut the measured shot budget by $2.67times$ ($1.33$--$4.00$) on hardware.
  arXiv  Detail & Related papers  (2026-09-13T10:47:03Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions [51.50375419691955]
  Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty.<n>We study how many samples are necessary and sufficient to learn an $varepsilon$-optimal robust policy under the average-reward criterion.
  arXiv  Detail & Related papers  (2026-08-06T19:49:48Z)
• Detecting and Discriminating Operator Misspecification in Hybrid PDE-Parameter Learning: a Reference-Free Instrument, with Discrimination Bounded In Sample [0.0]
  We build an instrument that reads from a single fit and with no oracle.<n>We report where the instrument is blind, a pre-registered negative where a neural estimator loses to Tikhonov-regularized inversion at recovery.
  arXiv  Detail & Related papers  (2026-08-04T21:20:44Z)
• Scaling Laws for Behavioral Foundation Models over User Event Sequences [2.924581427482972]
  We study a common two-part behavioral-model architecture, a feature-based event embedder, and a decoder-only transformer.<n>Across roughly 600 runs on real interaction data, spanning $1015$-$1019$ training FLOPs, we vary four deployment-relevant axes.<n> Compute-optimal training is data-heavy relative to text at low compute, but its $D/N$ ratio moves toward the Chinchilla as compute increases.
  arXiv  Detail & Related papers  (2026-06-03T15:59:25Z)
• Scaling Laws for Agent Harnesses via Effective Feedback Compute [53.68149869349268]
  emphEffective Feedback Compute (EFC) is a trace-level scaling coordinate that credits feedback only when it is informative, valid, non-redundant, and retained for subsequent decisions.<n>EFC-based coordinates consistently predict failure rates better than raw-compute baselines.
  arXiv  Detail & Related papers  (2026-05-28T09:45:47Z)
• Scale-Sensitive Shattering: Learnability and Evaluability at Optimal Scale [54.65053906803857]
  We study the optimal scale at which real-valued function classes exhibit uniform convergence and learnability.<n>Our main result establishes a scale-sensitive generalization of the fundamental theorem of PAC learning.<n>We also highlight several open questions on quantitative sample complexity and evaluability.
  arXiv  Detail & Related papers  (2026-05-13T15:41:30Z)
• Breaking the KV Cache Bottleneck: Fan Duality Model Achieves O(1) Decode Memory with Superior Associative Recall [0.0]
  Fan Duality Model resolves tension between memory efficiency and associative recall in sequence modeling.<n>FDM separates sequence processing into two components: a wave component that compresses long-range patterns into a hidden state, and a particle component that retrieves specific tokens.<n>We propose Freeze-Scan, a two-phase training strategy that freezes the recurrent scan and optimize the cache jointly with embeddings.
  arXiv  Detail & Related papers  (2026-04-09T02:00:30Z)
• Learn by Surprise, Commit by Proof [0.0]
  We propose a self-gated post-training framework for autonomous knowledge acquisition.<n>When a passage produces anomalously high pertoken loss, L SCP flags it, generates a Q&A chain that forces the model to articulate its own knowledge.<n>The entire learning intensity is governed by a single parameter $r$.
  arXiv  Detail & Related papers  (2026-04-02T12:17:10Z)
• QANTIS: A Hardware-Validated Quantum Platform for POMDP Planning and Multi-Target Data Association [4.11040996037715]
  We present QS, a modular platform that integrates quantum belief update (Grover amplitude amplification and BIQAE), QUBO-based data association via FPC-QAOA, and composable error mitigation.<n>On hardware, a single Grover iterate applied to a Tiger belief oracle amplifies a rare observation probability from $0.179$ to $9.07$ ($5.1times$; ISA 18) while preserving the Bayesian posterior.<n>We further demonstrate, to our knowledge, the first closed-loop hybrid quantum-classical Tiger POMDP on superconducting hardware.
  arXiv  Detail & Related papers  (2026-02-28T19:13:44Z)
• Beyond likelihood ratio bias: Nested multi-time-scale stochastic approximation for likelihood-free parameter estimation [49.78792404811239]
  We study inference in simulation-based models where the analytical form of the likelihood is unknown.<n>We use a ratio-free nested multi-time-scale approximation (SA) method that simultaneously tracks the score and drives the parameter update.<n>We show that our algorithm can eliminate the original bias $Obig(sqrtfrac1Nbig)$ and accelerate the convergence rate from $Obig(beta_k+sqrtfracalpha_kNbig)$.
  arXiv  Detail & Related papers  (2024-11-20T02:46:15Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• Robust Counterfactual Explanations for Neural Networks With Probabilistic Guarantees [11.841312820944774]
  We propose a measure -- that we call $textitStability$ -- to quantify the robustness of counterfactuals to potential model changes for differentiable models. Our main contribution is to show that counterfactuals with sufficiently high value of $textitStability$ will remain valid after potential model changes with high probability.
  arXiv  Detail & Related papers  (2023-05-19T20:48:05Z)
• Minimax Optimal Quantization of Linear Models: Information-Theoretic Limits and Efficient Algorithms [59.724977092582535]
  We consider the problem of quantizing a linear model learned from measurements. We derive an information-theoretic lower bound for the minimax risk under this setting. We show that our method and upper-bounds can be extended for two-layer ReLU neural networks.
  arXiv  Detail & Related papers  (2022-02-23T02:39:04Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• Estimation in Tensor Ising Models [5.161531917413708]
  We consider the problem of estimating the natural parameter of the $p$-tensor Ising model given a single sample from the distribution on $N$ nodes. In particular, we show the $sqrt N$-consistency of the MPL estimate in the $p$-spin Sherrington-Kirkpatrick (SK) model. We derive the precise fluctuations of the MPL estimate in the special case of the $p$-tensor Curie-Weiss model.
  arXiv  Detail & Related papers  (2020-08-29T00:06:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.