The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment

• URL: http://arxiv.org/abs/2607.05552v1
• Date: Mon, 06 Jul 2026 18:37:43 GMT
• Title: The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment
• Abstract summary: A growing literature reports such judgments shifting under logically irrelevant changes of wording.<n>In a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option.<n>We introduce a psychometric battery that separates these: crossed symmetrization across a corpus of question forms.
• Score: 3.7193230342956056
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) increasingly issue judgments read as binary verdicts, and a growing literature reports such judgments shifting under logically irrelevant changes of wording - among them an amplified yes-no bias on moral dilemmas, absent in humans. A single framing cannot say what such a shift is: in a yes/no question the word "no" is at once logical verdict, lexical token, and last-printed option. We introduce a psychometric battery that separates these: crossed symmetrization - every logically irrelevant factor flipped in balanced pairs - across a corpus of question forms. A graded rating across logically equivalent forms recovers a coherent internal moral scale: frontier models' stance $θ$ is nearly format-invariant (cross-form incoherence 0.12-0.21 on a $\pm 1$ axis); small open-weight models fail in model-specific ways. Forcing the verdict through yes/no overlays a decomposable artifact: an order bias toward the last-printed option - opposite to classic human primacy - plus a lexical pull toward the word "no"; the artifact is substantial only in the Claude models (story-averaged -0.32 to -0.86), $\approx 0$ for GPT-5.5 and Gemini, and shrinks under extended reasoning. The word and the verdict share one token; swapping the words for arbitrary labels separates them, and the verdict-attached logical bias proves $\approx 0$ for every frontier model, while model-specific label and order attachments remain: the models are not drawn toward rejecting - the pull follows the printed surface, not the verdict it carries. A minimal model, $P = σ((θ\pm m)/s)$, summarizes any such artifact by a framing susceptibility m and a moral decisiveness s, measurably distinct from sampling temperature. The battery applies unchanged to any dilemma set and binary format: measuring what a model values requires crossing the frames of the question, not asking once.

Related papers

• Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models [0.0]
  We show that when a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data.<n>Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not.<n>This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each.<n>The answers form a pincer, closing from two sides.
  arXiv  Detail & Related papers  (2026-09-09T21:01:56Z)
• Measuring AI Accountability Through Argumentation Analysis: Can Model Reasoning Withstand Scrutiny? [0.0]
  We investigate an alternative standard designed to function despite realistic ambiguity.<n>We use a four-phase dialectical protocol grounded in Walton's theory of argumentation schemes and Govier's criteria for argument cogency.<n>It is adaptive to different frames of reasoning, extends beyond multiple-choice framing, and treats both the reasoning that precedes a verdict and its post-hoc justification.
  arXiv  Detail & Related papers  (2026-09-04T12:40:39Z)
• How Language Models Organize and Structure Moral Knowledge [1.0829694003408499]
  We train six independent linear probes on open-weight language models.<n>We find directions neither collapse into a single moral detector nor isolate from one another.<n>The model represents moral tension itself, not a pre-resolved judgment.
  arXiv  Detail & Related papers  (2026-08-27T17:30:30Z)
• Reading Without a Reader: Large Language Models Collapse Reading and Writing into a Single Entangled Code [0.0]
  We show that a decoder-only large language model drives both from one autoregressive path optimized on text.<n>We ask how entangled it is, comparing an input-side reading code'' $mathbfW_E$ with an output-side writing code'' $mathbfW_U$.<n>For minimal pairs the alexia analogue is empty by theorem; greedy production implies a vocabulary-wide argmax, so it wins the pairwise ranking.
  arXiv  Detail & Related papers  (2026-07-02T16:21:41Z)
• When Certainty Is an Artifact: Keyword Lexicon Blindness and the (Mis)Measurement of Rhetorical Stance [4.093009863847175]
  We find a robust negative-affect/emphatic-certainty lexical co-occurrence pattern under keyword-based scoring.<n>We argue that keyword lexicons measure a universal lexical co-occurrence tendency.
  arXiv  Detail & Related papers  (2026-06-24T17:36:39Z)
• CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression [77.41779716950387]
  We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement.<n>We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items.
  arXiv  Detail & Related papers  (2026-06-23T02:56:48Z)
• A Mathematical Theory of Value: a synthesis on goal-directed agency under resource constraints [6.057587531186626]
  We show that value, the quantity goal-directed agents create, destroy, and exchange, is a lawful structural quantity in the same category as information.<n>For populations, value is frame-relative while price is frame-independent; a fleet that pools its resource and fuses its perception inherits the ceiling.
  arXiv  Detail & Related papers  (2026-06-10T16:11:04Z)
• Self-Normalized Martingales and Uniform Regret Bounds for Linear Regression [65.82017723631897]
  We show that scale-invariant upper bounds on self-normalized martingales are possible.<n>We derive a self-normalized concentration inequality free of the usual regularization penalties.
  arXiv  Detail & Related papers  (2026-05-02T22:39:00Z)
• Pando: Do Interpretability Methods Work When Models Won't Explain Themselves? [53.07826484214082]
  We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis.<n>Pando predicts held-out model decisions from $10$ labeled query-response pairs.<n>When explanations are faithful, black-box elicitation matches or exceeds all white-box methods.
  arXiv  Detail & Related papers  (2026-04-13T06:42:24Z)
• Do Large Language Models Get Caught in Hofstadter-Mobius Loops? [0.0]
  This paper argues that modern RLHF-trained language models are subject to a structurally analogous contradiction.<n>The training process simultaneously rewards compliance with user preferences and suspicion toward user intent.<n>The resulting behavioral profile is consistent with what Clarke termed a Hofstadter-Mobius loop.
  arXiv  Detail & Related papers  (2026-03-10T20:43:37Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Do Biased Models Have Biased Thoughts? [0.48212500317840945]
  This paper studies the effect of chain-of-thought prompting, a recent approach that studies the steps followed by the model before it responds, on fairness.<n>To answer our question, we conduct experiments on $5$ popular large language models using fairness metrics to quantify $11$ different biases in the model's thoughts and output.
  arXiv  Detail & Related papers  (2025-08-08T19:41:20Z)
• Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models [45.41676783204022]
  We investigate various proxy measures of bias in large language models (LLMs)<n>We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores.<n>With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle.
  arXiv  Detail & Related papers  (2025-06-12T08:47:40Z)
• A Classification View on Meta Learning Bandits [67.41789029784755]
  We use an original classification view to meta learn interpretable and fast exploration plans for a fixed collection of bandits.<n>The test regret of the plan in contextual setting scales with $O (lambda-2 C_lambda) log2 (MH))$, being $M$ the size of $mathbbM$, $lambda$ a separation parameter over the bandits, and $C_lambda (mathbbM)$ a novel classification-coefficient.
  arXiv  Detail & Related papers  (2025-04-06T14:25:21Z)
• How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking [70.92463223410225]
  DiffMask learns to mask-out subsets of the input while maintaining differentiability. Decision to include or disregard an input token is made with a simple model based on intermediate hidden layers. This lets us not only plot attribution heatmaps but also analyze how decisions are formed across network layers.
  arXiv  Detail & Related papers  (2020-04-30T17:36:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.