Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random

• URL: http://arxiv.org/abs/2609.00576v1
• Date: Tue, 01 Sep 2026 02:12:10 GMT
• Title: Consistency Without Alignment: Item-Sensitive Language Models Indistinguishable From Random
• Abstract summary: Item-sensitivity, defined as whether a model's choice depends on the specific input rather than on its own output prior, is widely reported as evidence of task competence.<n>We show this evidence is necessary but not sufficient using a forced-choice signalling task abstracted from the board game Deception: Murder in Hong Kong.<n>Across seven language models, two model families, a post-training ablation, and three independent scoring rules, every one of 21 model-by-rule cells is reliably item-sensitive.<n>We call this consistency without alignment and argue it generalises to any evaluation that relies on item-sensitivity, permutation consistency
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Item-sensitivity, defined as whether a model's choice depends on the specific input rather than on its own output prior, is widely reported as evidence of task competence. We show this evidence is necessary but not sufficient using a forced-choice signalling task abstracted from the board game Deception: Murder in Hong Kong. In this environment, the reference points against which a coordinate should be judged (a fit-maximising strategy, a posterior-maximising strategy, and uniform random selection) are all computable in closed form. Across seven language models, two model families, a post-training ablation, and three independent scoring rules, every one of 21 model-by-rule cells is reliably item-sensitive. Yet 8 of those 21 cells are not statistically distinguishable from a chooser that ignores the item and selects at random, and 5 score worse than random at describing the target. Item-sensitivity and distance from random correlate at only r = 0.30. We call this consistency without alignment and argue it generalises to any evaluation that relies on item-sensitivity, permutation consistency, or self-consistency without an independent reference for the measured quantity. We further find that a literal-similarity baseline with no pragmatics outperforms most tested language models, that adding a pragmatic layer over two baseline similarity sources moves choosers toward random rather than toward the Bayesian reference, and that a standard labelled multiple-choice format carries no measurable content signal here. All results represent the model side of a pre-registered instrument; a matched human condition is designed and piloted but not yet collected.

Related papers

• Predictive Likelihood Ratios for Language Model Watermark Detection [3.8200540264474907]
  Keyed watermark detection tests dependence between observed tokens and pseudorandom variables reconstructed from a secret key.<n>We construct predictive likelihood ratios that average over uncertain probability deficits and residual-tail distributions.
  arXiv  Detail & Related papers  (2026-09-14T14:39:29Z)
• Off-Target Effects of Response-Style Alignment in a Korean 27B Language Model [2.8678936590409028]
  We measure two behaviours the objective never targets: abstention on ambiguous social questions and unprompted disclosure in securities guidance.<n>Matched target-form controls show that answer propensity depends on the training target, not the prompt set or recipe alone.
  arXiv  Detail & Related papers  (2026-09-10T09:23:04Z)
• Accuracy and Order Sensitivity Diverge Under Label-Free Strategies [4.127762845164668]
  Multiple-choice benchmarks are widely used to evaluate large language models.<n>Multiple choice scores conflate knowledge with sensitivity to option order.<n>In this paper, we test whether preventing a model from seeing option labels while committing to an answer removes positional influence.
  arXiv  Detail & Related papers  (2026-08-12T11:34:45Z)
• The Matching Principle: A Geometric Theory of Loss Functions for Nuisance-Robust Representation Learning [0.5854803320592717]
  We prove optimality in a linear-Gaussian model (Thm. A), necessity of range coverage for any quadratic penalty that zeros deployment drift (Thm. G)<n>Thirteen blocks (ML through Qwen2.5-7B) test matched vs isotropic vs wrong-direction penalties on geometry and deployment drift.<n>We claim a falsifiable design recipe: estimate Sigma_task, match Sigma', run the controls, report task and geometry separately.
  arXiv  Detail & Related papers  (2026-05-21T17:53:28Z)
• A Mechanistic Study of Tabular Foundation Models [12.26738662548555]
  Tabular foundation models converge in accuracy across a range of classification and regression tasks.<n>This raises questions a leaderboard cannot answer.<n>We characterize all three: (i) whether the models execute the same in-context algorithm, (ii) where row, column, and class-permutation invariances originate, and (iii) how robust they are under perturbations engineered against the inferred mechanism.
  arXiv  Detail & Related papers  (2026-05-20T15:23:16Z)
• EigenBench: A Comparative Behavioral Measure of Value Alignment [0.28707625120094377]
  EigenBench is a black-box method for benchmarking language models' values.<n>It is designed to quantify subjective traits for which reasonable judges may disagree on the correct label.<n>It can recover model rankings on the GPQA benchmark without access to objective labels.
  arXiv  Detail & Related papers  (2025-09-02T04:14:26Z)
• SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models [0.27309692684728604]
  Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels.<n>This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner.
  arXiv  Detail & Related papers  (2025-07-24T08:28:17Z)
• Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression [102.24287051757469]
  We study self-supervised covariance estimation in deep heteroscedastic regression.<n>We derive an upper bound on the 2-Wasserstein distance between normal distributions.<n>Experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.
  arXiv  Detail & Related papers  (2025-02-14T22:37:11Z)
• Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts [119.22672589020394]
  We propose COnfidence-baSed MOdel Selection (CosMoS), where model confidence can effectively guide model selection. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift.
  arXiv  Detail & Related papers  (2023-06-19T18:48:15Z)
• Robust Outlier Rejection for 3D Registration with Variational Bayes [70.98659381852787]
  We develop a novel variational non-local network-based outlier rejection framework for robust alignment. We propose a voting-based inlier searching strategy to cluster the high-quality hypothetical inliers for transformation estimation.
  arXiv  Detail & Related papers  (2023-04-04T03:48:56Z)
• Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation [63.77513071533095]
  Most existing sequential methods assume users are deterministic. Item-item transitions might fluctuate significantly in several item aspects and exhibit randomness of user interests. We propose a Distribution-based Transformer Sequential Recommendation (DT4SR) which injects uncertainties into sequential modeling.
  arXiv  Detail & Related papers  (2021-06-11T04:35:21Z)
• Consensus-Guided Correspondence Denoising [67.35345850146393]
  We propose to denoise correspondences with a local-to-global consensus learning framework to robustly identify correspondence. A novel "pruning" block is introduced to distill reliable candidates from initial matches according to their consensus scores estimated by dynamic graphs from local to global regions. Our method outperforms state-of-the-arts on robust line fitting, wide-baseline image matching and image localization benchmarks by noticeable margins.
  arXiv  Detail & Related papers  (2021-01-03T09:10:00Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.