Language Models Generalize to Human-like Word Order Preferences

• URL: http://arxiv.org/abs/2608.05028v1
• Date: Wed, 05 Aug 2026 16:33:46 GMT
• Title: Language Models Generalize to Human-like Word Order Preferences
• Abstract summary: We investigate whether language models exhibit the same bias under similar conditions.<n>We find that they consistently prefer scope-homomorphic orders despite never observing them during training.<n>While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences.
• Score: 1.9371675344367494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central question in language acquisition is whether linguistic biases can emerge from general learning mechanisms operating over underdetermined input. Artificial Language Learning (ALL) studies have shown that human learners reliably generalize beyond the evidence provided, including by preferring scope-homomorphic noun phrase modifier orders. In this work, we investigate whether language models exhibit the same bias under similar conditions. We create a controlled learning environment in which models are trained on a corpus where all noun phrases containing multiple modifiers have been removed, eliminating direct evidence about modifier ordering, and are then evaluated on multiple modifier sentences. Across three model sizes, we find that they consistently prefer scope-homomorphic orders despite never observing them during training. These preferences vary in strength by modifier type. To investigate the source of these preferences, we examine noun-modifier association strength using pointwise mutual information (PMI). While PMI reflects known modifier-ordering patterns, it does not explain the models' ordering preferences. These findings demonstrate that LMs can recover human-like linguistic generalizations from impoverished input and provide a controlled framework for investigating the mechanisms underlying such biases.

Related papers

• Isolating LLM Lexical Bias: A Curation-Free Triangulated Metric for Preference-Stage Learning [0.33463386076921503]
  We introduce the Triangulated Preference Shift score, a metric that triangulates between human gold standards, base models, and variants.<n>We provide data across six model families, anchor the results in the literature, and illustrate the general approach's utility.
  arXiv  Detail & Related papers  (2026-05-29T20:19:49Z)
• Learning inflection classes using Adaptive Resonance Theory [0.2676349883103403]
  We study the learnability of a system of verbal inflection classes by the individual language user.<n>We use Adaptive Resonance Theory, a neural network with a parameter that determines the degree of generalisation (vigilance)<n>The similarity of clustering to attested inflection classes varies depending on the complexity of the inflectional system.
  arXiv  Detail & Related papers  (2025-12-17T15:58:20Z)
• DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models [50.54264918467997]
  Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks.<n>Recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language.<n>We propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior.
  arXiv  Detail & Related papers  (2025-02-25T16:44:10Z)
• Can Language Models Learn Typologically Implausible Languages? [62.823015163987996]
  Grammatical features across human languages show intriguing correlations often attributed to learning biases in humans.<n>We discuss how language models (LMs) allow us to better determine the role of domain-general learning biases in language universals.<n>We test LMs on an array of highly naturalistic but counterfactual versions of the English (head-initial) and Japanese (head-final) languages.
  arXiv  Detail & Related papers  (2025-02-17T20:40:01Z)
• Black Big Boxes: Do Language Models Hide a Theory of Adjective Order? [5.395055685742631]
  In English and other languages, multiple adjectives in a complex noun phrase show intricate ordering patterns that have been a target of much linguistic theory. We review existing hypotheses designed to explain Adjective Order Preferences (AOPs) in humans and develop a setup to study AOPs in language models. We find that all models' predictions are much closer to human AOPs than predictions generated by factors identified in theoretical linguistics.
  arXiv  Detail & Related papers  (2024-07-02T10:29:09Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers [54.4919139401528]
  We show that it is possible to reduce interference by identifying and pruning language-specific parameters. We show that removing identified attention heads from a fixed model improves performance for a target language on both sentence classification and structural prediction.
  arXiv  Detail & Related papers  (2022-10-11T18:11:37Z)
• Examining the Inductive Bias of Neural Language Models with Artificial Languages [42.699545862522214]
  We propose a novel method for investigating the inductive biases of language models using artificial languages. This constitutes a fully controlled causal framework, and demonstrates how grammar engineering can serve as a useful tool for analyzing neural models.
  arXiv  Detail & Related papers  (2021-06-02T09:34:32Z)
• Unnatural Language Inference [48.45003475966808]
  We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words. Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
  arXiv  Detail & Related papers  (2020-12-30T20:40:48Z)
• Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model [66.84264870118723]
  We present the first purely corpus-driven model of multi-lingual adjective ordering in the form of a latent-variable model. We provide strong converging evidence for the existence of universal, cross-linguistic, hierarchical adjective ordering tendencies.
  arXiv  Detail & Related papers  (2020-10-09T18:27:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.