Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models

• URL: http://arxiv.org/abs/2510.12110v1
• Date: Tue, 14 Oct 2025 03:26:28 GMT
• Title: Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models
• Authors: Ziliang Qiu, Renfen Hu,
• Abstract summary: We propose PACE, asking LLMs to generate Association Chains to evaluate their creativity.<n> PACE minimizes the risk of data contamination and offers a straightforward, highly efficient evaluation.
• Score: 0.3580891736370874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The evaluation of LLMs' creativity represents a crucial research domain, though challenges such as data contamination and costly human assessments often impede progress. Drawing inspiration from human creativity assessment, we propose PACE, asking LLMs to generate Parallel Association Chains to Evaluate their creativity. PACE minimizes the risk of data contamination and offers a straightforward, highly efficient evaluation, as evidenced by its strong correlation with Chatbot Arena Creative Writing rankings (Spearman's $\rho = 0.739$, $p < 0.001$) across various proprietary and open-source models. A comparative analysis of associative creativity between LLMs and humans reveals that while high-performing LLMs achieve scores comparable to average human performance, professional humans consistently outperform LLMs. Furthermore, linguistic analysis reveals that both humans and LLMs exhibit a trend of decreasing concreteness in their associations, and humans demonstrating a greater diversity of associative patterns.

Related papers

• On Evaluating LLM Alignment by Evaluating LLMs as Judges [68.15541137648721]
  evaluating large language models' (LLMs) alignment requires them to be helpful, honest, safe, and to precisely follow human instructions.<n>We examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences.<n>We propose a benchmark that assesses alignment without directly evaluating model outputs.
  arXiv  Detail & Related papers  (2025-11-25T18:33:24Z)
• A word association network methodology for evaluating implicit biases in LLMs compared to humans [0.0]
  We present a novel word association network methodology for evaluating implicit biases in Large language models (LLMs)<n>Our method taps into the implicit relational structures encoded in LLMs, providing both quantitative and qualitative assessments of bias.<n>To demonstrate the utility of our methodology, we apply it to both humans and several widely used LLMs to investigate social biases related to gender, religion, ethnicity, sexual orientation, and political party.
  arXiv  Detail & Related papers  (2025-10-28T15:03:18Z)
• A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models [100.16387798660833]
  Oogiri game is a creativity-driven task requiring humor and associative thinking.<n>LoTbench is an interactive, causality-aware evaluation framework.<n>Results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable.
  arXiv  Detail & Related papers  (2025-01-25T09:11:15Z)
• The LLM Effect: Are Humans Truly Using LLMs, or Are They Being Influenced By Them Instead? [60.01746782465275]
  Large Language Models (LLMs) have shown capabilities close to human performance in various analytical tasks. This paper investigates the efficiency and accuracy of LLMs in specialized tasks through a structured user study focusing on Human-LLM partnership.
  arXiv  Detail & Related papers  (2024-10-07T02:30:18Z)
• Divergent Creativity in Humans and Large Language Models [37.67363469600804]
  Large Language Models (LLMs) have led to claims that they are approaching a level of creativity akin to human capabilities.<n>We leverage recent advances in computational creativity to analyze semantic divergence in both state-of-the-art LLMs and a dataset of 100,000 humans.<n>We found evidence that LLMs can surpass average human performance on the Divergent Association Task, and approach human creative writing abilities.
  arXiv  Detail & Related papers  (2024-05-13T22:37:52Z)
• Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches [69.73783026870998]
  This work proposes a novel framework, ValueLex, to reconstruct Large Language Models' unique value system from scratch. Based on Lexical Hypothesis, ValueLex introduces a generative approach to elicit diverse values from 30+ LLMs. We identify three core value dimensions, Competence, Character, and Integrity, each with specific subdimensions, revealing that LLMs possess a structured, albeit non-human, value system.
  arXiv  Detail & Related papers  (2024-04-19T09:44:51Z)
• Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing [56.75702900542643]
  We introduce AlphaLLM for the self-improvements of Large Language Models.<n>It integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving loop.<n>Our experimental results show that AlphaLLM significantly enhances the performance of LLMs without additional annotations.
  arXiv  Detail & Related papers  (2024-04-18T15:21:34Z)
• Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions [6.201550639431176]
  This paper focuses on the burgeoning prominence of large language models (LLMs) We underscore the pivotal role of Reinforcement Learning from Human Feedback (RLHF) in augmenting LLMs' rationality and decision-making prowess.
  arXiv  Detail & Related papers  (2024-03-14T18:36:04Z)
• Assessing and Understanding Creativity in Large Language Models [33.37237667182931]
  This paper aims to establish an efficient framework for assessing the level of creativity in large language models (LLMs) By adapting the Torrance Tests of Creative Thinking, the research evaluates the creative performance of various LLMs across 7 tasks. We found that the creativity of LLMs primarily falls short in originality, while excelling in elaboration.
  arXiv  Detail & Related papers  (2024-01-23T05:19:47Z)
• Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks [65.69651759036535]
  We analyze whether large language models (LLMs) can serve as reliable alternatives to humans.<n>This paper explores both conventional tasks (e.g., story generation) and alignment tasks (e.g., math reasoning)<n>We find that LLM evaluators can generate unnecessary criteria or omit crucial criteria, resulting in a slight deviation from the experts.
  arXiv  Detail & Related papers  (2023-10-30T17:04:35Z)
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate [57.71597869337909]
  We build a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments.
  arXiv  Detail & Related papers  (2023-08-14T15:13:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.