Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models

• URL: http://arxiv.org/abs/2606.21645v1
• Date: Fri, 19 Jun 2026 17:56:44 GMT
• Title: Behavioral and Representational Evidence of Binomial Ordering Preferences in Large Language Models
• Abstract summary: We use linguistic binomials, such as men and women, where both word permutations are grammatically valid but exhibit distinct, cross-linguistic variations in conventionality.<n>We measure and compare the corpus-derived preferences with model-induced ordering probabilities of 6 open-weight language models.
• Score: 61.21623682203952
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) can readily reproduce conventional expressions, yet their ability to model gradient frequency distributions remains underexplored. We investigate this using linguistic binomials, such as men and women, where both word permutations are grammatically valid but exhibit distinct, cross-linguistic variations in conventionality. We formalize binomial ordering as a distributional alignment problem, and construct a multilingual dataset of 600 binomial pairs across 8 languages. With categorical and distributional metrics, we measure and compare the corpus-derived preferences with model-induced ordering probabilities of 6 open-weight LLMs. While models often behaviorally recover the dominant corpus-preferred order, particularly for strongly conventionalized pairs, they align less well with the exact corpus preference distributions. This suggests that apparent directional order overstates how faithfully LLMs capture the statistical nuances of language use. Sparse probing verifies that the concept of preference strength is partially encoded among middle-to-late layers, and steering along probe-derived directions alters model-induced ordering distributions, demonstrating that the statistical behavioral preference of LLMs can be mechanistically measured and manipulated via internal representations.

Related papers

• Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity [6.285055089721361]
  We introduce a principled interpretability framework based on token-level perplexity to test whether models rely on linguistically relevant cues.<n>Our method enables precise, hypothesis-driven analysis without relying on unstable feature-attribution techniques.
  arXiv  Detail & Related papers  (2026-03-31T08:00:55Z)
• Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective [60.45433515408158]
  We show that long Chain-of-Thought (CoT) serves as a decisive decision-maker for the top option but fails to function as a granular distribution calibrator for ambiguous tasks.<n>We observe a distinct "decoupled mechanism": while CoT improves distributional alignment, final accuracy is dictated by CoT content.
  arXiv  Detail & Related papers  (2026-01-06T16:26:40Z)
• Probabilistic Token Alignment for Large Language Model Fusion [100.30692772017238]
  Training large language models (LLMs) from scratch can yield models with unique functionalities and strengths, but it is costly and often leads to redundant capabilities.<n>A key challenge in existing model fusion is their dependence on manually predefined vocabulary alignment.<n>We propose the probabilistic token alignment method as a general and soft mapping for alignment, named as PTA-LLM.
  arXiv  Detail & Related papers  (2025-09-21T23:18:24Z)
• Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies [11.615194868239849]
  We prove sequence perplexity is invariant under any factorization, including forward, backward, or arbitrary permutations.<n>We retrain GPT-2 models across forward, backward, and arbitrary permuted orders on scientific text.
  arXiv  Detail & Related papers  (2025-05-13T16:52:19Z)
• Random-Set Large Language Models [4.308457163593758]
  Large Language Models (LLMs) are known to produce very high-quality tests and responses to our queries.<n>But how much can we trust this generated text?<n>We propose a novel Random-Set Large Language Model (RSLLM) approach which predicts finite random sets (belief functions) over the token space.
  arXiv  Detail & Related papers  (2025-04-25T05:25:27Z)
• Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers [14.797001158310092]
  We argue that distributional semantics models struggle with truth-conditional reasoning and symbolic processing. Contrary to expectations, we find that LLMs align more closely with human judgements on exact quantifiers versus vague ones.
  arXiv  Detail & Related papers  (2024-10-17T19:28:35Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)
• A Latent Space Theory for Emergent Abilities in Large Language Models [5.033924641692716]
  We show that languages are not created randomly but rather to communicate information. A strong association between languages and their underlying meanings, resulting in a sparse joint distribution. With the advent of LLMs trained on big data and large models, we can now precisely assess the marginal distribution of languages.
  arXiv  Detail & Related papers  (2023-04-19T20:45:01Z)
• Evaluating Distributional Distortion in Neural Language Modeling [81.83408583979745]
  A heavy-tail of rare events accounts for a significant amount of the total probability mass of distributions in language. Standard language modeling metrics such as perplexity quantify the performance of language models (LM) in aggregate. We develop a controlled evaluation scheme which uses generative models trained on natural data as artificial languages.
  arXiv  Detail & Related papers  (2022-03-24T01:09:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.