Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift

• URL: http://arxiv.org/abs/2607.17384v2
• Date: Tue, 21 Jul 2026 07:59:25 GMT
• Title: Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift
• Abstract summary: This paper provides an experimentally verified formal law for calculating the uplift that diversity of thought provides in Large Language Model ensembles.<n>We test the law on 767,520 inferences from ten open-weight models over two graduate-level science benchmarks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper provides an experimentally verified formal law for calculating the uplift that diversity of thought provides in Large Language Model (LLM) ensembles. From first principles, we derive an exact decomposition of LLM ensemble lift into rescue and damage masses, which yields a compact heuristic for calculating uplift. From this we extract the metrics which predict ensemble performance: an accuracy-adjusted correctness correlation, $φ_{\mathrm{adj}}$, together with the accuracy gap and collective accuracy of the pair. We test the law on 767,520 inferences from ten open-weight models over two graduate-level science benchmarks, together with a novel agentic cybersecurity benchmark in which each model conducts digital-forensics investigations by multi-turn tool use in a network-isolated sandbox (23,520 graded trials including abstentions); all votes are released openly. Calibrated once on SuperGPQA at a 40:60 vote split, the heuristic predicts lift on the calibration set with Spearman's $ρ=0.84$ and, with its coefficients frozen, transfers to two datasets never used in calibration ($ρ=0.51$ on GPQA Diamond and $0.84$ on the forensic tasks), whilst the measured swap mass tracks realised lift with $R^2\ge 0.96$ throughout. Raw $φ$ has almost no predictive power ($R^2\le 0.09$ throughout); the accuracy-adjusted $φ_{\mathrm{adj}}$ is markedly superior ($R^2=0.67$ on SuperGPQA), and the heuristic combining these metrics is the most stable pre-pooling predictor across the three datasets.

Related papers

• Invertible Logits Transformation for Accuracy-Preserving Post-Hoc Uncertainty Calibration [10.333478793903724]
  An ideal calibrator should correct nonlinear miscalibration, scale gracefully to large label spaces, and preserve the original predictions.<n>InvLT consistently outperforms a broad set of post-hoc baselines on standard calibration metrics.
  arXiv  Detail & Related papers  (2026-08-11T02:08:05Z)
• Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate [0.0]
  Mixed-precision quantization must decide which parts of a model to keep at higher precision.<n>We show that the loss from quantizing a set of layers can be reconstructed from per-layer or pairwise sensitivities measured in isolation.<n>We propose the coverage model $f(S)=cbigl (1-prod_iin S (1-a_i)bigr)$, which reproduces the measured variance profile of $f$ to within a few percent from its $L$ fitted break-rates.
  arXiv  Detail & Related papers  (2026-07-14T02:08:33Z)
• Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics [0.0]
  Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse.<n>We show that weight decay acts as a scalar empirical control parameter for these regimes.<n>We introduce online diagnostics, mean pairwise attentionhead cosine similarity and entropy standard deviation, that track training dynamics.
  arXiv  Detail & Related papers  (2026-05-19T19:48:40Z)
• Representation Without Reward: A JEPA Audit for LLM Fine-Tuning [1.2691047660244335]
  Joint-embedding predictive architectures (JEPAs) propose that a model should learn more useful abstractions when trained to predict latent representations rather than observed outputs.<n>For autoregressive language-model fine-tuning the principle entails a stricter requirement: the induced hidden-state geometry must reach the language-model head emphand improve the decoded task metric.<n>We test that requirement under a fixed Llama-3.2-1B-Instruct LoRA harness on natural-language-to-regex generation.
  arXiv  Detail & Related papers  (2026-05-14T20:27:32Z)
• High-Probability Convergence in Decentralized Stochastic Optimization with Gradient Tracking [69.90407799170687]
  We study high-probability (HP) convergence guarantees in decentralized optimization.<n>Our results establish the same conditions on the horizon as in achieving comparable times.
  arXiv  Detail & Related papers  (2026-04-30T22:45:21Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Singular Bayesian Neural Networks [1.2891210250935148]
  Bayesian neural networks promise calibrated uncertainty but require $O(mn)$ parameters for standard mean-field Gaussian posteriors.<n>We induce a posterior that is singular with respect to the Lebesgue measure, concentrating on the rank-$r$ manifold.<n>We derive PAC-Bayes generalization bounds whose complexity term scales as $sqrtr(m+n)$ instead of $sqrtm n$, and prove loss bounds that decompose the error into optimization and rank-induced bias.
  arXiv  Detail & Related papers  (2026-01-30T23:06:34Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  We conduct an unprecedented empirical investigation training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law.<n>Our estimated optima deviates from the global best performance found via exhaustive search by merely 0.094% on the test set.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)
• Contextual Combinatorial Bandits with Probabilistically Triggered Arms [55.9237004478033]
  We study contextual bandits with probabilistically triggered arms (C$2$MAB-T) under a variety of smoothness conditions. Under the triggering modulated (TPM) condition, we devise the C$2$-UC-T algorithm and derive a regret bound $tildeO(dsqrtT)$.
  arXiv  Detail & Related papers  (2023-03-30T02:51:00Z)
• Optimal Robust Linear Regression in Nearly Linear Time [97.11565882347772]
  We study the problem of high-dimensional robust linear regression where a learner is given access to $n$ samples from the generative model $Y = langle X,w* rangle + epsilon$ We propose estimators for this problem under two settings: (i) $X$ is L4-L2 hypercontractive, $mathbbE [XXtop]$ has bounded condition number and $epsilon$ has bounded variance and (ii) $X$ is sub-Gaussian with identity second moment and $epsilon$ is
  arXiv  Detail & Related papers  (2020-07-16T06:44:44Z)
• Extrapolation Towards Imaginary $0$-Nearest Neighbour and Its Improved Convergence Rate [13.985534521589257]
  We propose a novel multiscale $k$-NN (MS-$k$-NN) that extrapolates unweighted $k$-NN estimators from several $k ge 1$ values to $k=0$. Our method implicitly computes optimal real-valued weights that are adaptive to the query and its neighbour points.
  arXiv  Detail & Related papers  (2020-02-08T00:32:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.