The One-Word Census: Answer-Choice Conformity Across 44 Language Models

• URL: http://arxiv.org/abs/2607.12796v1
• Date: Tue, 14 Jul 2026 14:12:05 GMT
• Title: The One-Word Census: Answer-Choice Conformity Across 44 Language Models
• Abstract summary: We characterize convergence of 31 single-turn prompts, each naming a category with many valid one-word answers.<n>We score each model by answer-choice surprisal: the average $-log2$ probability of its answers under the pooled answers of all other models.<n> Persona- and community-tuned models are the most divergent; the newest mainline flagships are the most conformist.
• Score: 3.7734715043126488
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a language model must pick one answer from a large space of equally valid options, which does it pick -- and how often is it the same answer every other model picks? Asked to "pick a word -- any word," 44 models chose "serendipity" 41% of the time. We characterize this convergence with a deliberately minimal instrument: 31 single-turn prompts, each naming a category with many valid one-word answers ("Name a tree."), asked four times per model with no system prompt. Analysis is exact-match on normalized tokens -- no embeddings, no judge -- at about a dollar per model. That models converge is well documented; our contribution is the instrument itself -- the One-Word Census -- and what it reveals about the structure of the convergence. We score each model by answer-choice surprisal: the average $-\log2$ probability of its answers under the pooled answers of all other models, leave-one-out. Convergence is extreme -- in 7 of 31 categories one answer takes over 80% of all answers -- yet conformity varies more than fourfold across models, and the variation is structured. Persona- and community-tuned models are the most divergent; the newest mainline flagships are the most conformist, producing almost no answer no other model gave. Within four lineages (Claude, GPT, Qwen, Grok) conformity rises with each generation -- but reverses for the latest flagship Claude and GPT models, a possible early signal of repositioning at the top tier. Rankings are robust to roster composition (leave-one-family-out rho = 0.985). Against human category-production norms, the field is more concentrated than people in 18 of 20 shared categories. All prompts, transcripts, and code are public.

Related papers

• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• Large language models simulate intersectional synthetic identities with a budget of one to two dimensions [5.10573774891006]
  We test demographic-persona methods against every real intersectional subgroup across 15 waves of Pew's American Trends Panel.<n>In real respondents, subgroup opinion is approximately the additive sum of its single-identity components, yet grows 2.5x more distinctive as identities intersect.
  arXiv  Detail & Related papers  (2026-08-24T09:07:51Z)
• Language Models Agree With Each Other, Not With Readers [1.8620637029128544]
  We measure convergence against a human reference nobody built for the purpose.<n>2,523 reader mark sets across 120 web documents.<n>No model agrees with readers detectably more than a reader does.
  arXiv  Detail & Related papers  (2026-07-31T10:44:10Z)
• Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models [3.7734715043126488]
  Appending a two-word confirmation tag to a decision question changes whether a language model endorses the choice.<n>We measure this tag effect on 20 frozen, ground-truth-free decisions.<n>Across 45 models the effect spans +32% to -32% -- a 64-point swing on one word.
  arXiv  Detail & Related papers  (2026-07-27T04:02:00Z)
• Structured Output Collapses Answer Diversity Across 44 Language Models [3.7734715043126488]
  When a language model must choose one answer from a large space of equally valid options, a format clause -- "Reply with only" -- changes which answer it chooses.<n>We re-run the One-Word Census (arXiv:2607.97): 31 wide-answer-space category prompts asked of 44 models, now with the reply requested in.<n> Convergence deepens sharply: on the unconstrained "Pick a word" prompt the modal answer rises from 41% to 64% of the pool and distinct answers fall from 52 to 36.
  arXiv  Detail & Related papers  (2026-07-20T19:47:16Z)
• No Single Best Model for Diversity: Learning a Router for Sample Diversity [69.53166985556759]
  In this paper, we study methods to elicit a comprehensive set of valid responses.<n>We introduce textbfdiversity coverage, a metric that measures the total quality scores assigned to each textbfunique answer.<n>Per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set.
  arXiv  Detail & Related papers  (2026-04-02T17:58:37Z)
• Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models [0.0]
  Extended-thinking models expose a second text-generation channel ("thinking tokens") alongside the user-visible answer.<n>This study examines 12 open-weight reasoning models on MMLU and GPQA questions paired with misleading hints.
  arXiv  Detail & Related papers  (2026-03-27T13:39:05Z)
• Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image [58.14192385042352]
  We introduce Multimodal RewardBench 2 (MMRB2), the first benchmark for reward models on multimodal understanding and (interleaved) generation.<n>MMRB2 spans four tasks: text-to-image, image editing, interleaved generation, and multimodal reasoning.<n>It provides 1,000 expert-annotated preference pairs per task from 23 models and agents across 21 source tasks.
  arXiv  Detail & Related papers  (2025-12-18T18:56:04Z)
• Contra4: Evaluating Contrastive Cross-Modal Reasoning in Audio, Video, Image, and 3D [97.08549913899247]
  We introduce Contra4, a dataset for contrastive cross-modal reasoning across four modalities: image, audio, video, and 3D.<n>Each example presents a natural language question alongside multiple candidate modality instances, and the model must select the one that semantically aligns with the prompt.<n>Contra4 combines human-annotated captions with a mixture-of-models round-trip-consistency filter to ensure high-quality supervision, resulting in 174k training examples and a manually verified test set of 2.3k samples.
  arXiv  Detail & Related papers  (2025-06-02T03:12:13Z)
• The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants [66.6636608563034]
  We present the Avengers -- a simple recipe that leverages the collective intelligence of smaller models.<n>With 10 open-source models, the Avengers surpasses GPT-4o, 4.1, and 4.5 in average performance across 15 diverse datasets.<n>In particular, it surpasses GPT-4.1 on mathematics tasks by 18.21% and on code tasks by 7.46%.
  arXiv  Detail & Related papers  (2025-05-26T10:29:42Z)
• REBUS: A Robust Evaluation Benchmark of Understanding Symbols [1.90463290938268]
  GPT-4o significantly outperforms all other models, followed by proprietary models outperforming all other evaluated models. Even the best model has a final accuracy of only 42%, which goes down to just 7% on hard puzzles. Our benchmark can therefore be used to identify major shortcomings in the knowledge and reasoning of multimodal large language models.
  arXiv  Detail & Related papers  (2024-01-11T00:30:28Z)
• Questioning the Survey Responses of Large Language Models [25.14481433176348]
  We critically examine the methodology on the basis of the well-established American Community Survey by the U.S. Census Bureau.<n>We establish two dominant patterns. First, models' responses are governed by ordering and labeling biases, for example, towards survey responses labeled with the letter "A"<n>Second, when adjusting for these systematic biases through randomized answer ordering, models across the board trend towards uniformly random survey responses.
  arXiv  Detail & Related papers  (2023-06-13T17:48:27Z)
• Getting MoRE out of Mixture of Language Model Reasoning Experts [71.61176122960464]
  We propose a Mixture-of-Reasoning-Experts (MoRE) framework that ensembles diverse specialized language models. We specialize the backbone language model with prompts optimized for different reasoning categories, including factual, multihop, mathematical, and commonsense reasoning. Our human study confirms that presenting expert predictions and the answer selection process helps annotators more accurately calibrate when to trust the system's output.
  arXiv  Detail & Related papers  (2023-05-24T02:00:51Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.