CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

• URL: http://arxiv.org/abs/2608.07460v1
• Date: Fri, 07 Aug 2026 17:55:48 GMT
• Title: CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity
• Abstract summary: CreativeInstruct is an instruction-tuning method that teaches large language models to balance creative, base-model-like generations with the quality of post-trained models.<n>We show that CreativeInstruct matches or exceeds the diversity of both multi-model baselines and distilled variants of their outputs.<n>We also show the benefits of creative models as a substrate for RL: GRPO applied to a CreativeInstruct checkpoint improves by 4% on AMC and 5% points on MATH.
• Score: 67.74087963315213
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While post-training improves the capabilities of large language models (LLMs), it generally lowers their output diversity and creativity, negatively impacting tasks that explicitly require creativity (e.g., story generation) as well as those that require it implicitly, e.g., reinforcement learning (RL). We instead propose CreativeInstruct, a scalable instruction-tuning method that teaches LLMs to balance creative, base-model-like generations with the quality of post-trained models, by learning to inject special [StartCreativity] spans that bias generation toward creativity. Furthermore, we introduce a structural diversity metric based on graph edit distance, which captures narrative level variation missed by purely lexical and semantic metrics. On narrative generation, CreativeInstruct matches or exceeds the diversity of both multi-model baselines and distilled variants of their outputs, without sacrificing quality or requiring multiple models at inference time. These results are mirrored in our human evaluation, where we find that annotators rate CreativeInstruct generations as more creative than the post-trained LLMs' generations in 70.3% of cases. We also show the benefits of creative models as a substrate for RL: GRPO applied to a CreativeInstruct checkpoint improves by ~4% on AMC and ~5% points on MATH over the same training applied to the post-trained checkpoint.

Related papers

• Playing with Words, Improving with Rewards: Training Language Models for Creative Association [9.232251624834644]
  Large Language Models (LLMs) are being applied to increasingly difficult problems and use cases.<n>To navigate their vast solution spaces effectively, LLMs need to be creative.<n>We train LLMs on Codenames, a word-association game that exercises the two central axes of creativity.
  arXiv  Detail & Related papers  (2026-05-27T01:41:22Z)
• CREATE: Testing LLMs for Associative Creativity [51.64544603712446]
  We introduce CREATE, a benchmark designed to evaluate models' capacity for creative associative reasoning.<n>We show that the strongest models achieve higher creative utility than others.<n>Our results illustrate that thinking models are not always more effective on our task, even with high token budgets.
  arXiv  Detail & Related papers  (2026-03-10T17:58:44Z)
• Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models [6.036586911740041]
  Large language models (LLMs) are increasingly used in verbal creative tasks.<n>The widely used Divergent Association Task ( DAT) focuses on novelty, ignoring appropriateness.<n>We evaluate a range of state-of-the-art LLMs on DAT and show that their scores on the task are lower than those of two baselines that do not possess any creative abilities.
  arXiv  Detail & Related papers  (2026-01-28T12:41:32Z)
• CreativityPrism: A Holistic Benchmark for Large Language Model Creativity [64.18257552903151]
  Creativity is often seen as a hallmark of human intelligence.<n>There is still no holistic framework to evaluate their creativity across diverse scenarios.<n>We propose CreativityPrism, an evaluation analysis framework that decomposes creativity into three dimensions: quality, novelty, and diversity.
  arXiv  Detail & Related papers  (2025-10-23T00:22:10Z)
• Jointly Reinforcing Diversity and Quality in Language Model Generations [64.72289248044514]
  Post-training of Large Language Models (LMs) often prioritizes accuracy and helpfulness at the expense of diversity.<n>We address this challenge with Diversity-Aware Reinforcement Learning (DARLING), a framework that jointly optimize for response quality and semantic diversity.
  arXiv  Detail & Related papers  (2025-09-02T17:38:47Z)
• Creative Preference Optimization [17.849454438382995]
  We propose a novel alignment method that injects signals from multiple creativity dimensions into the preference optimization objective in a modular fashion.<n>We train and evaluate creativity-augmented versions of several models using CrPO and MuCE, a new large-scale human preference dataset.<n>Our models outperform strong baselines, including GPT-4o, on both automated and human evaluations.
  arXiv  Detail & Related papers  (2025-05-20T14:43:41Z)
• Cooking Up Creativity: Enhancing LLM Creativity through Structured Recombination [46.79423188943526]
  We introduce a novel approach that enhances Large Language Models (LLMs) creativity.<n>We apply LLMs for translating between natural language and structured representations, and perform the core creative leap.<n>We demonstrate our approach in the culinary domain with DishCOVER, a model that generates creative recipes.
  arXiv  Detail & Related papers  (2025-04-29T11:13:06Z)
• Modifying Large Language Model Post-Training for Diverse Creative Writing [12.872333448726595]
  In creative writing generation, we investigate post-training approaches to promote both output diversity and quality.<n>Our core idea is to include deviation in the training objective to facilitate learning from rare high-quality instances.<n>Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models.
  arXiv  Detail & Related papers  (2025-03-21T13:21:45Z)
• Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM [58.42678619252968]
  Creation-MMBench is a benchmark designed to evaluate the creative capabilities of Multimodal Large Language Models.<n>The benchmark comprises 765 test cases spanning 51 fine-grained tasks.<n> Experimental results reveal that open-source MLLMs significantly underperform compared to proprietary models in creative tasks.
  arXiv  Detail & Related papers  (2025-03-18T17:51:34Z)
• A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models [100.16387798660833]
  Oogiri game is a creativity-driven task requiring humor and associative thinking.<n>LoTbench is an interactive, causality-aware evaluation framework.<n>Results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable.
  arXiv  Detail & Related papers  (2025-01-25T09:11:15Z)
• Creativity Has Left the Chat: The Price of Debiasing Language Models [1.223779595809275]
  We investigate the unintended consequences of Reinforcement Learning from Human Feedback on the creativity of Large Language Models (LLMs) Our findings have significant implications for marketers who rely on LLMs for creative tasks such as copywriting, ad creation, and customer persona generation.
  arXiv  Detail & Related papers  (2024-06-08T22:14:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.