GENIE: A Fine-Grained Measure for Novelty

• URL: http://arxiv.org/abs/2606.12790v1
• Date: Thu, 11 Jun 2026 01:14:10 GMT
• Title: GENIE: A Fine-Grained Measure for Novelty
• Abstract summary: Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks.<n>Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner.<n>We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features.
• Score: 52.134780556216974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks. Prior work has focused on addressing whether models are capable of generating creative outputs. Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner. We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features with respect to a population of responses. We show that unlike GENIE, holistic metrics struggle to capture the high-dimensionality of novelty and do not provide insight on which properties they target. Finally, we use GENIE to measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty.

Related papers

• Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks [6.696727578876306]
  We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed.<n>We develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments.<n>Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance.
  arXiv  Detail & Related papers  (2026-08-30T21:21:08Z)
• Automated Creativity Evaluation of Language Models Across Open-Ended Tasks [3.6948012022450776]
  Large language models (LLMs) have achieved remarkable progress in language understanding, reasoning, and generation.<n>We introduce an automated, domain-agnostic framework for quantifying LLM creativity across open-ended tasks.<n>Our approach separates the measurement apparatus from the creative task itself, enabling scalable, task-agnostic assessment.
  arXiv  Detail & Related papers  (2026-06-10T07:37:06Z)
• InnoGym: Benchmarking the Innovation Potential of AI Agents [74.64144272881414]
  InnoGym is the first benchmark designed to evaluate the innovation potential of AI agents.<n>InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches.
  arXiv  Detail & Related papers  (2025-12-01T16:03:04Z)
• CreativityPrism: A Holistic Benchmark for Large Language Model Creativity [64.18257552903151]
  Creativity is often seen as a hallmark of human intelligence.<n>There is still no holistic framework to evaluate their creativity across diverse scenarios.<n>We propose CreativityPrism, an evaluation analysis framework that decomposes creativity into three dimensions: quality, novelty, and diversity.
  arXiv  Detail & Related papers  (2025-10-23T00:22:10Z)
• Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity [29.58419742230708]
  N-gram novelty is widely used to evaluate language models' ability to generate text outside of their training data.<n>We investigate the relationship between this notion of creativity and n-gram novelty through close reading of human and AI-generated text.<n>We find that while n-gram novelty is positively associated with expert writer-judged creativity, 91% of top-quartile expressions by n-gram novelty are not judged as creative.
  arXiv  Detail & Related papers  (2025-09-26T17:59:05Z)
• Measuring LLM Novelty As The Frontier Of Original And High-Quality Output [22.876599736420832]
  We introduce a new novelty metric for large language models (LLMs) generations that balances originality and quality.<n>We find that model-generated text from some base LLMs is less novel than human-written text from the internet.
  arXiv  Detail & Related papers  (2025-04-13T00:48:58Z)
• Learning Task Representations from In-Context Learning [67.66042137487287]
  Large language models (LLMs) have demonstrated remarkable proficiency in in-context learning (ICL)<n>We introduce an automated formulation for encoding task information in ICL prompts as a function of attention heads.<n>The proposed method successfully extracts task-specific information from in-context demonstrations and excels in both text and regression tasks.
  arXiv  Detail & Related papers  (2025-02-08T00:16:44Z)
• The Creative Frontier of Generative AI: Managing the Novelty-Usefulness Tradeoff [0.4873362301533825]
  We explore the optimal balance between novelty and usefulness in generative Artificial Intelligence (AI) systems. Overemphasizing either aspect can lead to limitations such as hallucinations and memorization.
  arXiv  Detail & Related papers  (2023-06-06T11:44:57Z)
• Deliberate then Generate: Enhanced Prompting Framework for Text Generation [70.10319005141888]
  Deliberate then Generate (DTG) prompting framework consists of error detection instructions and candidates that may contain errors. We conduct extensive experiments on 20+ datasets across 7 text generation tasks, including summarization, translation, dialogue, and more. We show that DTG consistently outperforms existing prompting methods and achieves state-of-the-art performance on multiple text generation tasks.
  arXiv  Detail & Related papers  (2023-05-31T13:23:04Z)
• A Unified Evaluation Framework for Novelty Detection and Accommodation in NLP with an Instantiation in Authorship Attribution [25.52598351435189]
  We introduce 'NoveltyTask', a multi-stage task to evaluate a system's performance on pipelined novelty 'detection' and 'accommodation' tasks. We use Amazon reviews corpus and compile a large dataset (consisting of 250k instances across 200 authors/labels) for NoveltyTask.
  arXiv  Detail & Related papers  (2023-05-08T22:37:30Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.