Language Models Agree With Each Other, Not With Readers

• URL: http://arxiv.org/abs/2607.29274v1
• Date: Fri, 31 Jul 2026 10:44:10 GMT
• Title: Language Models Agree With Each Other, Not With Readers
• Abstract summary: We measure convergence against a human reference nobody built for the purpose.<n>2,523 reader mark sets across 120 web documents.<n>No model agrees with readers detectably more than a reader does.
• Score: 1.8620637029128544
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Claims that language models homogenise are usually measured against human judgements collected for the study, which makes the human side an artifact of the design: a crowdworker given the model's instruction is running the model's prompt. We measure convergence against a human reference nobody built for the purpose -- 2,523 reader mark sets across 120 web documents, produced by people highlighting for their own reasons on a platform where the overlay of others' marks is off by default. Agreement is the overlap between two size-matched sentence sets minus the overlap expected when each is resampled within its own depth-and-length bands. The null's calibration is demonstrated, not asserted: every pair involving a random baseline lands within 0.006 of zero. On the median document each party names 14 sentences of 70; two readers share 4.1 and two models 8.7. Across 18 model arms spanning 11 vendors, 3 countries and both weight regimes, the median of 153 model pairs is +0.093 against a human yardstick of +0.040, and 99 sit entirely above the human interval. Two frontier models from rival labs reach +0.203, twice what GPT-4o agrees with itself on a second call. The effect is not determinism, prompt wording, procedure, vendor or routing, and it is graded: the smallest models agree at the human level. No model agrees with readers detectably more than a reader does, and at equal depth and length no surface feature separates their choices. The multiples are procedure-dependent and the ordering is not: models are cut to their sharpest set while a reader's is a random draw from what they marked, and blunting the models alike halves the gap without closing it. Tested out of sample on four models released after this analysis, against predictions fixed beforehand, none clears the human interval. A population simulated from several models is not several populations.

Related papers

• Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models [0.0]
  We show that when a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data.<n>Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not.<n>This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each.<n>The answers form a pincer, closing from two sides.
  arXiv  Detail & Related papers  (2026-09-09T21:01:56Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• How much of a measured AI preference is the model, and how much is the instrument? [0.0]
  This study holds the outcomes and the models fixed and varies the instrument alone.<n>To conclude, a preference obtained from one instrument carries little information about what a second instrument would report.
  arXiv  Detail & Related papers  (2026-08-24T02:23:11Z)
• Matched Outcomes, Divergent Gaze: How Foveated MLLMs Search Compared to Humans [50.71222402701153]
  We compare three general-purpose large language models (MLLMs) with human eye-movement scanpaths on goal-directed search (COCO-Search18)<n>All three share one signature: low-entropy, large-amplitude, self-consistent scanpaths that agree with themselves far more closely than two humans agree with each other.<n>The gap sits on a process axis that answer-alignment and saliency metrics do not measure. Because they miss it, such metrics cannot certify human-like vision, and zero-shot models suit outcome and spatial questions but not temporal, process-level
  arXiv  Detail & Related papers  (2026-08-17T12:54:39Z)
• Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents [22.30468215041597]
  A tool-using agent is given the same task in a different language, does it still take the same steps?<n>We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages.
  arXiv  Detail & Related papers  (2026-08-11T16:18:34Z)
• When AI Writes, Who Gets Cited? Evidence of Citation Monoculture Across Language Models [45.91367912142692]
  We show that different models may still select the same narrow subset, producing citation monoculture without any single citation being wrong.<n>Eleven models from three vendors choose at most ten papers from uniformly random panels of thirty, with real titles and abstracts but fabricated authors, reassigned years, and hidden venues and citation counts.
  arXiv  Detail & Related papers  (2026-08-03T14:27:46Z)
• OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment [0.9821874476902969]
  We introduce OptimismBench, which detects directional bias with inverted pairs.<n>We show post-training sets the sign of the bias, with opposite shifts in different families.<n>We release 3,870 items across 10 languages for per-model directional-bias auditing.
  arXiv  Detail & Related papers  (2026-07-29T14:38:55Z)
• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models [0.0]
  Two experiments cross all three factors on one held, contamination-free synthetic corpus.<n>Experiment 1 measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement.<n>Experiment 2 measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder.
  arXiv  Detail & Related papers  (2026-07-21T16:31:35Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• The One-Word Census: Answer-Choice Conformity Across 44 Language Models [3.7734715043126488]
  We characterize convergence of 31 single-turn prompts, each naming a category with many valid one-word answers.<n>We score each model by answer-choice surprisal: the average $-log2$ probability of its answers under the pooled answers of all other models.<n> Persona- and community-tuned models are the most divergent; the newest mainline flagships are the most conformist.
  arXiv  Detail & Related papers  (2026-07-14T14:12:05Z)
• The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel [0.0]
  I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does.<n>I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full.
  arXiv  Detail & Related papers  (2026-06-22T08:48:50Z)
• The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking [0.0]
  This paper presents an eight-week observational comparison of 68 single-file HTML generations collected across 17 public experiments in the "HTML AI Battle" project.<n>Four reasoning model families, GPT, Gemini, Grok, and Claude, were compared under a fixed public-interface protocol with no custom instructions, no personality tuning, and no repair prompts.<n> Claude was the strongest and most consistent family, leading mean performance and winning 9/17 prompts under the primary human weighted score.
  arXiv  Detail & Related papers  (2026-05-06T18:19:11Z)
• Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models [45.41676783204022]
  We investigate various proxy measures of bias in large language models (LLMs)<n>We find that evaluating models with pre-prompted personae on a multi-subject benchmark (MMLU) leads to negligible and mostly random differences in scores.<n>With the recent trend for LLM assistant memory and personalization, these problems open up from a different angle.
  arXiv  Detail & Related papers  (2025-06-12T08:47:40Z)
• Model Equality Testing: Which Model Is This API Serving? [59.005869726179455]
  API providers may quantize, watermark, or finetune the underlying model, changing the output distribution.<n>We formalize detecting such distortions by Model Equality Testing, a two-sample testing problem.<n>A test built on a simple string kernel achieves a median of 77.4% power against a range of distortions.
  arXiv  Detail & Related papers  (2024-10-26T18:34:53Z)
• One Thousand and One Pairs: A "novel" challenge for long-context language models [56.60667988954638]
  NoCha is a dataset of 1,001 pairs of true and false claims about 67 fictional books. Our annotators confirm that the largest share of pairs in NoCha require global reasoning over the entire book to verify. On average, models perform much better on pairs that require only sentence-level retrieval vs. global reasoning.
  arXiv  Detail & Related papers  (2024-06-24T02:03:57Z)
• What Can We Learn from Collective Human Opinions on Natural Language Inference Data? [88.90490998032429]
  ChaosNLI is a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS. This dataset is created by collecting 100 annotations per example for 3,113 examples in SNLI and MNLI and 1,532 examples in Abductive-NLI.
  arXiv  Detail & Related papers  (2020-10-07T17:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.