Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks

• URL: http://arxiv.org/abs/2607.20864v1
• Date: Thu, 23 Jul 2026 02:45:02 GMT
• Title: Position Bias is Hidden Behind Ceiling Effects: A Permutation Diagnostic for LLM Benchmarks
• Abstract summary: I introduce inspect_permute, an open-source extension to inspect_ai- evaluation framework.<n>It runs exhaustive answer_order permutations per question.<n>It reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Position bias in multiple-choice LLM evaluation is widely cited as a confound in capability comparisons, but published measurements rely on single answer-order shuffles whose results confound the bias signal with content-level noise and sampling stochasticity. I introduce inspect_permute, an open-source extension to the inspect_ai evaluation framework that runs exhaustive answer-order permutations per question and reports the chi-squared / Cramer V signature of position bias with bootstrap confidence intervals. I apply the tool across four vendors (gpt-4o-mini, claude-haiku-4-5, gemini-2.5-flash, grok-3) on five MMLU subjects, 24,000 API calls under temperature-0 generation, with falsifier predictions pre-registered via a public SHA-256 hash before half the data was observed. Position bias turns out to be statistically detectable only within a roughly 60-95% base-accuracy Goldilocks zone. Below it, processing-load dominance swamps subject-specific signal; above it, ceiling effects compress the variance below the chi-squared test resolution. Detectable cells separate into two mechanism types: monotone A-to-D decrease (processing_load, in low-tier models) and non-monotone D-drop (content_ambiguity, in a narrow capability band). Standard MMLU places every frontier-tier model above the detection band, so absence of signal there should be read as not measurable, not unbiased. Together with the ceiling-effect characterisation in arXiv:2606.26185, this work brackets the detectable region of position-bias measurement and makes the field central question askable in a verifiable form. Package, data, preregistration under MIT.

Related papers

• GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models [19.055058751178056]
  Generative vision-language models (VLMs) are increasingly used in human-centered settings.<n>They can produce demographically biased outputs even when images differ only in controlled attributes such as perceived race or gender.<n>We propose GGSS---Geodesic-Gated Spherical Steering---a norm-preserving intervention that steers visual tokens along geodesic arcs.
  arXiv  Detail & Related papers  (2026-08-26T04:58:40Z)
• Change Detection in Probability Flow ODE: Online Testing in Diffusion Latent Spaces [0.0]
  A growing range of sequential data tasks, such as identifying trend reversals in financial markets, cannot be fully addressed without detection of distributional shifts in time-ordered data.<n>We consider a sequential change-point detection problem where the conditional density switches at an unknown time, yet neither the pre- nor post-change distribution admits a closed-form.<n>A conditional diffusion model, trained on pre-change-point data with a frozen context encoder, defines a deterministicection via the probability flow ODE.
  arXiv  Detail & Related papers  (2026-08-24T05:09:24Z)
• Laplace-Fisher Gate Identities for Optimal Matrix-Gated Blended Score Estimation [0.0]
  Laplace-Fisher Gate Identity (LFGI)<n> finite-reference LFGI estimator to normalized density evaluation for Bayesian inverse problems.<n>Experiments using LFGI with known model evidence check absolute evidence calibration in both Gaussian and non-Gaussian settings.
  arXiv  Detail & Related papers  (2026-06-23T21:00:32Z)
• Anticipating the Optimism Gap: Predicting Distribution-Shift Degradation of RF-Impairment Detectors from In-Distribution Statistics [0.0]
  Detectors for radio-frequency impairments are usually reported with a single AUC measured on the distribution they were tuned on.<n>We ask whether this optimism can be predicted before any out-of-distribution data is seen.
  arXiv  Detail & Related papers  (2026-06-20T14:10:59Z)
• PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading [0.0]
  PlotChain is a deterministic, generator-based benchmark for evaluating large language models (MLLMs)<n>PlotChain contains 15 plot families with 450 rendered plots (30 per family)<n>Top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%.
  arXiv  Detail & Related papers  (2026-01-29T06:08:19Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Benchmarking Fraud Detectors on Private Graph Data [70.4654745317714]
  Currently, many types of fraud are managed in part by automated detection algorithms that operate over graphs.<n>We consider the scenario where a data holder wishes to outsource development of fraud detectors to third parties.<n>Third parties submit their fraud detectors to the data holder, who evaluates these algorithms on a private dataset and then publicly communicates the results.<n>We propose a realistic privacy attack on this system that allows an adversary to de-anonymize individuals' data based only on the evaluation results.
  arXiv  Detail & Related papers  (2025-07-30T03:20:15Z)
• CBW: Towards Dataset Ownership Verification for Speaker Verification via Clustering-based Backdoor Watermarking [85.68235482145091]
  Large-scale speech datasets have become valuable intellectual property.<n>We propose a novel dataset ownership verification method.<n>Our approach introduces a clustering-based backdoor watermark (CBW)<n>We conduct extensive experiments on benchmark datasets, verifying the effectiveness and robustness of our method against potential adaptive attacks.
  arXiv  Detail & Related papers  (2025-03-02T02:02:57Z)
• Less is More: Fewer Interpretable Region via Submodular Subset Selection [54.07758302264416]
  This paper re-models the above image attribution problem as a submodular subset selection problem. We construct a novel submodular function to discover more accurate small interpretation regions. For correctly predicted samples, the proposed method improves the Deletion and Insertion scores with an average of 4.9% and 2.5% gain relative to HSIC-Attribution.
  arXiv  Detail & Related papers  (2024-02-14T13:30:02Z)
• Proximity-Informed Calibration for Deep Neural Networks [49.330703634912915]
  ProCal is a plug-and-play algorithm with a theoretical guarantee to adjust sample confidence based on proximity. We show that ProCal is effective in addressing proximity bias and improving calibration on balanced, long-tail, and distribution-shift settings.
  arXiv  Detail & Related papers  (2023-06-07T16:40:51Z)
• MAPS: A Noise-Robust Progressive Learning Approach for Source-Free Domain Adaptive Keypoint Detection [76.97324120775475]
  Cross-domain keypoint detection methods always require accessing the source data during adaptation. This paper considers source-free domain adaptive keypoint detection, where only the well-trained source model is provided to the target domain.
  arXiv  Detail & Related papers  (2023-02-09T12:06:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.