Measuring diversity of synthetic prompts and data generated with fine-grained persona prompting

• URL: http://arxiv.org/abs/2505.17390v1
• Date: Fri, 23 May 2025 02:00:00 GMT
• Title: Measuring diversity of synthetic prompts and data generated with fine-grained persona prompting
• Authors: Gauri Kambhatla, Chantal Shaib, Venkata Govindarajan,
• Abstract summary: We measure the diversity of persona-driven synthetically generated prompts and responses with a suite of lexical diversity and redundancy metrics.<n>We find that synthetic prompts are significantly less diverse than human-written ones.<n>While persona-prompting does improve lexical diversity (especially with larger models), fine-grained detail in personas doesn't increase diversity noticeably.
• Score: 2.773884499834578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-grained personas have recently been used for generating 'diverse' synthetic data for pre-training and supervised fine-tuning of Large Language Models (LLMs). In this work, we measure the diversity of persona-driven synthetically generated prompts and responses with a suite of lexical diversity and redundancy metrics. Firstly, we find that synthetic prompts/instructions are significantly less diverse than human-written ones. Next, we sample responses from LLMs of different sizes with fine-grained and coarse persona descriptions to investigate how much fine-grained detail in persona descriptions contribute to generated text diversity. We find that while persona-prompting does improve lexical diversity (especially with larger models), fine-grained detail in personas doesn't increase diversity noticeably.

Related papers

• A Penalty Goes a Long Way: Measuring Lexical Diversity in Synthetic Texts Under Prompt-Influenced Length Variations [21.27593629875137]
  Penalty-Adjusted Type-Token Ratio (PATTR) is a diversity metric robust to length variations.<n>We generate a large synthetic corpus of over 20M words using seven models from the LLaMA, OLMo, and Phi families.
  arXiv  Detail & Related papers  (2025-07-20T19:14:43Z)
• Evaluating the Diversity and Quality of LLM Generated Content [72.84945252821908]
  We introduce a framework for measuring effective semantic diversity--diversity among outputs that meet quality thresholds.<n>Although preference-tuned models exhibit reduced lexical and syntactic diversity, they produce greater effective semantic diversity than SFT or base models.<n>These findings have important implications for applications that require diverse yet high-quality outputs.
  arXiv  Detail & Related papers  (2025-04-16T23:02:23Z)
• Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning [23.456302461693053]
  Possibility Exploration Fine-Tuning (PEFT) is a task-agnostic framework that enhances the text diversity of Large Language Models (LLMs) without increasing latency or computational cost.<n>PEFT significantly enhances the diversity of LLM outputs, as evidenced by lower similarity between candidate responses.<n>It can also notably reduce demographic bias in dialogue systems.
  arXiv  Detail & Related papers  (2024-12-04T14:23:16Z)
• Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting [28.971248570622603]
  We propose a diversity metric called structural diversity, where the user provides a mapping from generated text to features capturing the kinds of diversity that they care about. In our experiments, we show that for structural diversity in the poetry and code domains, CoS significantly improves diversity compared to several baselines.
  arXiv  Detail & Related papers  (2024-08-12T14:34:06Z)
• Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment [84.32768080422349]
  Alignment with human preference prevents large language models from generating misleading or toxic content. We propose a new formulation of prompt diversity, implying a linear correlation with the final performance of LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2024-03-17T07:08:55Z)
• Improving Demonstration Diversity by Human-Free Fusing for Text-to-SQL [51.48239006107272]
  In this paper, we discuss how to measure and improve the diversity of the demonstrations for text-to-diversity research. We propose fusing iteratively for demonstrations (Fused) to build a high-diversity demonstration pool. Our method achieves an average improvement of 3.2% and 5.0% with and without human labeling on several mainstream datasets.
  arXiv  Detail & Related papers  (2024-02-16T13:13:18Z)
• Exploring Diversity in Back Translation for Low-Resource Machine Translation [85.03257601325183]
  Back translation is one of the most widely used methods for improving the performance of neural machine translation systems. Recent research has sought to enhance the effectiveness of this method by increasing the 'diversity' of the generated translations. This work puts forward a more nuanced framework for understanding diversity in training data, splitting it into lexical diversity and syntactic diversity.
  arXiv  Detail & Related papers  (2022-06-01T15:21:16Z)
• Random Network Distillation as a Diversity Metric for Both Image and Text Generation [62.13444904851029]
  We develop a new diversity metric that can be applied to data, both synthetic and natural, of any type. We validate and deploy this metric on both images and text.
  arXiv  Detail & Related papers  (2020-10-13T22:03:52Z)
• MGD-GAN: Text-to-Pedestrian generation through Multi-Grained Discrimination [96.91091607251526]
  We propose the Multi-Grained Discrimination enhanced Generative Adversarial Network, that capitalizes a human-part-based Discriminator and a self-cross-attended Discriminator. A fine-grained word-level attention mechanism is employed in the HPD module to enforce diversified appearance and vivid details. The substantial improvement over the various metrics demonstrates the efficacy of MGD-GAN on the text-to-pedestrian synthesis scenario.
  arXiv  Detail & Related papers  (2020-10-02T12:24:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.