Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging

• URL: http://arxiv.org/abs/2604.26206v1
• Date: Wed, 29 Apr 2026 01:23:34 GMT
• Title: Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
• Abstract summary: A predecessor pilot found that Llama-3-8B prompted implements sandbagging as positional collapse rather than answer avoidance.<n>This follow-up added cyclic option-order randomisation as the critical control.<n> Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) added cyclic option-order randomisation as the critical control. The pre-registered item-level same-letter diagnostic did not confirm deterministic position-tracking (same-letter rate 37.3%, below the 50% threshold). However, pre-specified supporting analyses revealed that the response-position distribution under sandbagging was highly stable under complete content rotation (Pearson r = 0.9994; Jensen-Shannon divergence = 0.027, compared to 0.386 between honest and sandbagging conditions). Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A. The data provide strong evidence for a soft distributional attractor: under sandbagging instruction, the model enters a low-entropy response-position basin centred on E/F/G that is highly stable and largely content-invariant at the aggregate level. Qwen-2.5-7B served as a negative control (non-compliant, no distributional shift). These results provide evidence, at the 7-9 billion parameter scale, that response-position entropy is a promising black-box behavioural signature of this sandbagging mode.

Related papers

• SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation [22.56867206175605]
  SCOPE-OPSD projects the privileged teacher-student residual onto a frozen rank-64 factor.<n>It reuses the forwards already required by OPSD and adds neither rollouts nor inference-time modules.<n>Across the complete 25/50/75/100-step trajectories for Qwen3-1.7B, 4B, and 8B, Structured is never below Pure OPSD.
  arXiv  Detail & Related papers  (2026-09-11T08:33:52Z)
• Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores [31.709549159768727]
  We observe a consistent readout gap across diverse reasoning benchmarks.<n> hidden-state probes successfully decode correct answers even when native sequence scoring completely collapses.<n>Our results show that many apparent zero-shot reasoning deficits are expression failures masking intact internal logic.
  arXiv  Detail & Related papers  (2026-08-31T16:43:55Z)
• Cross-Fitted Residual Utility for Primary-Preserving Cognitive Decision Correction in Automatic Modulation Classification [1.547549252134621]
  We study a post-inference problem through cross-fitted residual utility and a primary-preserving cognitive decision policy.<n>A structured KAN-Fourier classifier supplies the default probability, while neural and non-neural candidates provide observable evidence.<n>On RMLA, RMLB, and HISAR, the complete system improves overall accuracy from 63.632% to 66.332%, 65.161% to 66.168%, and 77.769% to 79.867%.
  arXiv  Detail & Related papers  (2026-08-03T11:07:37Z)
• Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning [33.56197101721029]
  Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization.<n>The resulting staleness is an inevitable byproduct, compounded jointly by policy lag, engine delays, and mixture-of-experts routing.<n>We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy.
  arXiv  Detail & Related papers  (2026-07-21T05:27:50Z)
• Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers [0.0]
  We present an online monitoring system for distributional shift in deployed safety classifiers.<n>After detection, a conformal abstention layer adapts decision to recover a target error.<n>The system achieves 86.6% detection with mean latency of 39.5800 steps.
  arXiv  Detail & Related papers  (2026-06-10T11:24:25Z)
• Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model [47.71489969727656]
  Conformalized Quantile Regression (CQR) delivers the best coverage in the bin with the poorest model predictions.<n>Only the Locally Valid and Discriminative (LVD) framework provides finite-sample emphlocal validity.
  arXiv  Detail & Related papers  (2026-06-05T18:29:19Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Adaptive Consensus in LLM Ensembles via Sequential Evidence Accumulation: Automatic Budget Identification and Calibrated Commit Signals [0.0]
  DASE produces a commit-type routing partition that generalises across benchmarks.<n> Adaptive stopping, not injection bandwidth, drives accuracy.<n>Injection-based methods exhibit an inverted-U accuracy-vs-inference trajectory.
  arXiv  Detail & Related papers  (2026-05-05T19:24:10Z)
• Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance [0.0]
  We tested whether symptom validity testing logic could identify sandbagging through below-chance performance (BCB) on forced-choice items.<n>We found zero of 12 model-domain cells showed significant below-chance performance under sandbagging instruction.<n>We propose that positional-distribution shift may be a more effective behavioural signature than below-chance accuracy for detecting prompted underperformance at this model scale.
  arXiv  Detail & Related papers  (2026-04-28T05:57:23Z)
• Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation [86.62036852878354]
  Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding.<n>We propose RoPE-Perturbed Self-Distillation, a training regularizer that improves positional robustness.<n> Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks.
  arXiv  Detail & Related papers  (2026-04-15T18:46:35Z)
• The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation [1.8345614451086532]
  RLHF-aligned language models exhibit response homogenization on TruthfulQA.<n>40-79% of questions produce a single semantic cluster across 10 i.i.d. samples.
  arXiv  Detail & Related papers  (2026-03-25T09:35:15Z)
• Entropy trajectory shape predicts LLM reasoning reliability: A diagnostic study of uncertainty dynamics in chain-of-thought [0.0]
  We study whether the shape of uncertainty dynamics across reasoning steps--captured by sampling a few answer completions per step--predicts correctness.<n>We introduce entropy-trajectory monotonicity: a chain is monotone if its per-step answer-distribution entropy decreases every step.
  arXiv  Detail & Related papers  (2026-03-19T14:17:16Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Bayesian Transformer for Probabilistic Load Forecasting in Smart Grids [0.0]
  This study proposes a Bayesian Transformer framework that integrates three complementary uncertainty mechanisms into a PatchTST backbone.<n>A seven-level multi-quantile pinball-loss prediction head and post-training isotonic regression calibration produce sharp, near-nominally covered prediction intervals.<n>On the primary benchmark (PJM, H=24h), BT achieves a CRPS of 0.0289, improving 7.4% over Deep Ensembles and 29.9% over the deterministic LSTM.
  arXiv  Detail & Related papers  (2026-03-09T02:39:51Z)
• Evidence-based Distributional Alignment for Large Language Models [58.65469623911573]
  We propose Evi-DA, an evidence-based alignment technique that improves the fidelity and robustness of LLM-based distribution estimation.<n>Given a target country, Evi-DA retrieves related World Values Survey items and their answer distributions, predicts a coarse Welzel value signature for each option, and infers the country-conditioned answer distribution in a structured format.
  arXiv  Detail & Related papers  (2026-03-03T03:34:06Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.