A Comparative Approach to Assessing Linguistic Creativity of Large Language Models and Humans

• URL: http://arxiv.org/abs/2507.12039v2
• Date: Thu, 17 Jul 2025 15:27:29 GMT
• Title: A Comparative Approach to Assessing Linguistic Creativity of Large Language Models and Humans
• Authors: Anca Dinu, Andra-Maria Florescu, Alina Resceanu,
• Abstract summary: The test consists of various tasks aimed at assessing their ability to generate new original words and phrases.<n>We administered the test to 24 humans and to an equal number of LLMs, and we automatically evaluated their answers using OCSAI tool for three criteria: Originality, Elaboration, and Flexibility.<n>The results show that LLMs not only outperformed humans in all the assessed criteria, but did better in six out of the eight test tasks.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The following paper introduces a general linguistic creativity test for humans and Large Language Models (LLMs). The test consists of various tasks aimed at assessing their ability to generate new original words and phrases based on word formation processes (derivation and compounding) and on metaphorical language use. We administered the test to 24 humans and to an equal number of LLMs, and we automatically evaluated their answers using OCSAI tool for three criteria: Originality, Elaboration, and Flexibility. The results show that LLMs not only outperformed humans in all the assessed criteria, but did better in six out of the eight test tasks. We then computed the uniqueness of the individual answers, which showed some minor differences between humans and LLMs. Finally, we performed a short manual analysis of the dataset, which revealed that humans are more inclined towards E(extending)-creativity, while LLMs favor F(ixed)-creativity.

Related papers

• LMLPA: Language Model Linguistic Personality Assessment [11.599282127259736]
  Large Language Models (LLMs) are increasingly used in everyday life and research. measuring the personality of a given LLM is currently a challenge. This paper introduces the Language Model Linguistic Personality Assessment (LMLPA), a system designed to evaluate the linguistic personalities of LLMs.
  arXiv  Detail & Related papers  (2024-10-23T07:48:51Z)
• HLB: Benchmarking LLMs' Humanlikeness in Language Use [2.438748974410787]
  We present a comprehensive humanlikeness benchmark (HLB) evaluating 20 large language models (LLMs) We collected responses from over 2,000 human participants and compared them to outputs from the LLMs in these experiments. Our results reveal fine-grained differences in how well LLMs replicate human responses across various linguistic levels.
  arXiv  Detail & Related papers  (2024-09-24T09:02:28Z)
• Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition [38.822535662755314]
  We propose a sample-efficient human evaluation method for large language models (LLMs)<n>Our method automatically and adaptively selects a compact set of input instructions that maximize semantic discrepancy between pairs of LLM responses.<n>Human evaluators then perform three-alternative forced choices on these paired responses, which are aggregated into a global ranking using Elo rating.
  arXiv  Detail & Related papers  (2024-04-10T01:26:24Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)
• Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models [2.7013338932521416]
  We advocate for the revival of vocabulary tests as a valuable tool for assessing Large Language Models (LLMs) performance. We evaluate seven LLMs using two vocabulary test formats across two languages and uncover surprising gaps in their lexical knowledge.
  arXiv  Detail & Related papers  (2023-10-23T08:45:12Z)
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark [14.922083834969323]
  Large language models (LLMs) have been shown to perform well at a variety of syntactic, discourse, and reasoning tasks. We introduce a new theory-driven benchmark, SocKET, that contains 58 NLP tasks testing social knowledge.
  arXiv  Detail & Related papers  (2023-05-24T09:21:06Z)
• Can Large Language Models Be an Alternative to Human Evaluations? [80.81532239566992]
  Large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided. We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation.
  arXiv  Detail & Related papers  (2023-05-03T07:28:50Z)
• Can ChatGPT Assess Human Personalities? A General Evaluation Framework [70.90142717649785]
  Large Language Models (LLMs) have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored. This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests.
  arXiv  Detail & Related papers  (2023-03-01T06:16:14Z)
• Benchmarking Large Language Models for News Summarization [79.37850439866938]
  Large language models (LLMs) have shown promise for automatic summarization but the reasons behind their successes are poorly understood. We find instruction tuning, and not model size, is the key to the LLM's zero-shot summarization capability.
  arXiv  Detail & Related papers  (2023-01-31T18:46:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.