How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework

• URL: http://arxiv.org/abs/2605.23651v2
• Date: Tue, 26 May 2026 06:10:05 GMT
• Title: How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework
• Abstract summary: We propose a context-aware evaluation framework in which human-likeness is assessed.<n>We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber.
• Score: 0.47954782871066737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While factual correctness and task-performance have been in focus of Large Language Model (LLM) research for a long time, the fundamental question of how human-like generated texts are on a linguistic level has been underexplored. From a corpus-linguistic perspective, language production is inherently context-dependent, with distinct communicative contexts giving rise to differences in frequencies and co-occurrence patterns of linguistic features. A text failing to adhere to these patterns can be content-wise correct, but still be unfavorable to human readers. In this work, we propose a context-aware evaluation framework in which human-likeness is assessed using a two-sample problem between the linguistic feature distribution of a human reference corpus for a given register and a corresponding LLM-generated corpus. We implement this framework using the Maximum Mean Discrepancy (MMD) and the 67 lexico-grammatical features introduced by Biber, which are commonly applied in corpus linguistics. In our experiments, we compare seven instruction-tuned, open-source models across five English-language datasets spanning distinct registers against a human baseline. While across all tested setups, LLMs deviate from the human baseline, which models are closest to human language depends on the register and is not dictated by model size.

Related papers

• Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS) [3.71430272682278]
  We introduce the Word Coverage Score (WCS), a metric that quantifies the extent to which contextually appropriate human vocabulary is mathematically pruned by standard sampling filters.<n>By auditing open-weight models on human-authored corpus fragments, we identify which logical lexical choices are rendered unreachable by the decoder.
  arXiv  Detail & Related papers  (2026-05-26T16:44:25Z)
• Benchmarking Concept-Spilling Across Languages in LLMs [7.577675422356702]
  Large Language Models (LLMs) exhibit remarkable cross-lingual abilities, yet often exhibit a systematic bias toward representations from other languages.<n>This paper presents a novel comparative framework for evaluating multilingual semantic robustness by measuring how models handle polysemous words across languages.
  arXiv  Detail & Related papers  (2026-01-18T19:28:26Z)
• Computational Turing Test Reveals Systematic Differences Between Human and AI Language [0.0]
  Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior.<n>Existing validation efforts rely heavily on human-judgment-based evaluations.<n>This paper introduces a computational Turing test to assess how closely LLMs approximate human language.
  arXiv  Detail & Related papers  (2025-11-06T08:56:37Z)
• Can Language Models Learn Typologically Implausible Languages? [62.823015163987996]
  Grammatical features across human languages show intriguing correlations often attributed to learning biases in humans.<n>We discuss how language models (LMs) allow us to better determine the role of domain-general learning biases in language universals.<n>We test LMs on an array of highly naturalistic but counterfactual versions of the English (head-initial) and Japanese (head-final) languages.
  arXiv  Detail & Related papers  (2025-02-17T20:40:01Z)
• Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models [16.82812708514889]
  Code-switching, the phenomenon of alternating between two or more languages in a single conversation, presents unique challenges for Natural Language Processing (NLP) Most existing research focuses on either syntactic constraints or neural generation, with few efforts to integrate linguistic theory with large language models (LLMs) for generating natural code-switched text. We introduce EZSwitch, a novel framework that combines Equivalence Constraint Theory (ECT) with LLMs to produce linguistically valid and fluent code-switched text.
  arXiv  Detail & Related papers  (2024-10-30T03:03:32Z)
• From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition [6.617999710257379]
  We propose a three-stage framework to assess the abilities of LMs. We evaluate the generative capacities of LMs using methods from linguistic research.
  arXiv  Detail & Related papers  (2024-10-17T06:31:49Z)
• HLB: Benchmarking LLMs' Humanlikeness in Language Use [2.438748974410787]
  We present a comprehensive humanlikeness benchmark (HLB) evaluating 20 large language models (LLMs) We collected responses from over 2,000 human participants and compared them to outputs from the LLMs in these experiments. Our results reveal fine-grained differences in how well LLMs replicate human responses across various linguistic levels.
  arXiv  Detail & Related papers  (2024-09-24T09:02:28Z)
• CLSE: Corpus of Linguistically Significant Entities [58.29901964387952]
  We release a Corpus of Linguistically Significant Entities (CLSE) annotated by experts. CLSE covers 74 different semantic types to support various applications from airline ticketing to video games. We create a linguistically representative NLG evaluation benchmark in three languages: French, Marathi, and Russian.
  arXiv  Detail & Related papers  (2022-11-04T12:56:12Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.