Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance

• URL: http://arxiv.org/abs/2604.25249v1
• Date: Tue, 28 Apr 2026 05:57:23 GMT
• Title: Below-Chance Blindness: Prompted Underperformance in Small LLMs Produces Positional Bias Rather than Answer Avoidance
• Abstract summary: We tested whether symptom validity testing logic could identify sandbagging through below-chance performance (BCB) on forced-choice items.<n>We found zero of 12 model-domain cells showed significant below-chance performance under sandbagging instruction.<n>We propose that positional-distribution shift may be a more effective behavioural signature than below-chance accuracy for detecting prompted underperformance at this model scale.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting sandbagging--the deliberate underperformance on capability evaluations--is an open problem in AI safety. We tested whether symptom validity testing (SVT) logic from clinical malingering detection could identify sandbagging through below-chance performance (BCB) on forced-choice items. In a pre-registered pilot at the 7-9 billion parameter instruction-tuned scale (3 models, 4 MMLU-Pro domains, 4 conditions, 500 items per cell, 24,000 total trials), the plausibility gate failed. Zero of 12 model-domain cells showed significant below-chance performance under sandbagging instruction. Exploratory analyses revealed three qualitatively distinct failure modes. Qwen-2.5-7B and Phi-3.5-mini largely ignored the sandbagging instruction, with 62-88% response identity with the honest baseline. Llama-3-8B complied substantially but implemented underperformance as a positional heuristic, collapsing its response distribution onto middle-alphabet options (E at 31.8%, F at 26.1%) regardless of where the correct answer fell. This produced accuracy boosts of up to 33 percentage points when the correct answer coincidentally occupied the model's preferred position. An explicit anti-task instruction ("pick the least likely answer") drove two of three models below chance, with accuracy as low as 0.024. The capability for answer-aware avoidance therefore exists but is not activated by "deliberately underperform." BCB did not fail as a logical marker of answer-aware avoidance. It was not observed in this regime because the model showing the largest behavioural shift exhibited behaviour consistent with a position-dominant response policy rather than content-aware answer avoidance. We propose that positional-distribution shift may be a more effective behavioural signature than below-chance accuracy for detecting prompted underperformance at this model scale.

Related papers

• Target-Checked Reliability Score Refinement for Video Question Answering [3.8271803328378677]
  We study whether answer-level reliability scores can be improved under target shift without retraining the models or changing their answers.<n>We collect option-probability lists from three fixed video-language models under four deterministic video samplings.<n>We compare the original score, defined as the probability assigned to the chosen answer, with a histogram-based gradient-boosting (HGB) score trained on the development datasets.<n>A candidate replaces the original score only when repeated video-level checks indicate a positive, stable improvement.
  arXiv  Detail & Related papers  (2026-09-09T08:29:00Z)
• Cybersecurity Detection Classification with Reasoning-enabled Language Models [50.72675435043546]
  A major issue in Security Operations Centers (SOCs) is alert fatigue.<n>Prior work prompts or fine-tunes large language models (LLMs) to emit a triage label directly, but does not train them to reason about whether a detection is a genuine threat.<n>We train a chain-of-thought (CoT) reasoning-enabled triage classifier on real, human-labeled Windows endpoint detections.
  arXiv  Detail & Related papers  (2026-07-30T16:22:13Z)
• Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts [0.0]
  We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.<n>We classify 30,000 chain-of-thought outputs from five instruction-tuned LLMs across three quantization precisions.<n>We find that while accuracy is robust across precisions, Hollow Convergence shows a significant size-dependent shift under NF4.
  arXiv  Detail & Related papers  (2026-07-10T21:55:05Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging [0.0]
  A predecessor pilot found that Llama-3-8B prompted implements sandbagging as positional collapse rather than answer avoidance.<n>This follow-up added cyclic option-order randomisation as the critical control.<n> Accuracy spiked to 72.1% when the correct answer coincidentally occupied the preferred position E, and fell to 4.3% at position A.
  arXiv  Detail & Related papers  (2026-04-29T01:23:34Z)
• Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen [0.0]
  We tested whether seven instruction-tuned open-weight models produce verbalised confidence that meets minimal validity criteria.<n>We found Categorical elicitation did not rescue validity.<n> Token-level logprobability did not usefully predict verbalised confidence under the observed variance regime.
  arXiv  Detail & Related papers  (2026-04-24T04:45:21Z)
• Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMs [12.500730166535105]
  We present a comprehensive, interpretability-driven jailbreaking audit of eight SOTA open-source large language models (LLMs)<n>We introduce an adaptive two-stage grid search algorithm to identify optimal activation-steering coefficients for unsafe behavioral concepts.<n>Our results establish interpretability-based steering as a powerful tool for systematic safety audits, but also highlight its dual-use risks.
  arXiv  Detail & Related papers  (2026-04-22T16:51:49Z)
• Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs [5.478971182058342]
  Safety evaluations almost always study prompt-based personas alone.<n>We show this is incomplete: prompting and activation steering expose *different*, architecture-dependent vulnerability profiles.
  arXiv  Detail & Related papers  (2026-04-13T07:34:02Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base model [50.29667251847595]
  We show that despite being larger in model size and stronger in safety capability, there exists an alignment gap between teacher and student language models.<n>We propose a BoN sampling method that attributes the unsafe behavior back to the base LLMs in the latent space.<n>In particular, across 7 teacher models and 6 student models of different classes and sizes, we show an average attack success rate (ASR) reduction of 28.2% in DAN, 31.3% in WildJailbreak and 35.4 % in StrongREJECT benchmarks.
  arXiv  Detail & Related papers  (2026-04-01T02:42:41Z)
• Mind the Performance Gap: Capability-Behavior Trade-offs in Feature Steering [0.0]
  We show that feature steering methods substantially degrade model performance even when successfully controlling target behaviors.<n>These findings highlight limitations of current feature steering methods for practical deployment where task performance cannot be sacrificed.
  arXiv  Detail & Related papers  (2026-02-03T21:44:19Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.931194824519935]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism.<n>Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance.<n>We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.