Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting

• URL: http://arxiv.org/abs/2408.06186v1
• Date: Mon, 12 Aug 2024 14:34:06 GMT
• Title: Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting
• Authors: Halley Young, Yimeng Zeng, Jacob Gardner, Osbert Bastani,
• Abstract summary: We propose a diversity metric called structural diversity, where the user provides a mapping from generated text to features capturing the kinds of diversity that they care about. In our experiments, we show that for structural diversity in the poetry and code domains, CoS significantly improves diversity compared to several baselines.
• Score: 28.971248570622603
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The capability to generate diverse text is a key challenge facing large language models (LLMs). Thus far, diversity has been studied via metrics such as $n$-gram diversity or diversity of BERT embeddings. However, for these kinds of diversity, the user has little control over the dimensions along which diversity is considered. For example, in the poetry domain, one might desire diversity in terms of rhyme and meter, whereas in the code domain, one might desire diversity in terms of the kinds of expressions used to solve a problem. We propose a diversity metric called structural diversity, where the user provides a mapping from generated text to features capturing the kinds of diversity that they care about. In addition, we propose a novel strategy called chain-of-specification (CoS) prompting for improving diversity by first having the LLM generate a specification encoding one instance of structural features, and then prompting the LLM to generate text that satisfies these features; notably, our strategy works with blackbox LLMs. In our experiments, we show that for structural diversity in the poetry and code domains, CoS significantly improves diversity compared to several baselines.

Related papers

• GANPrompt: Enhancing Robustness in LLM-Based Recommendations with GAN-Enhanced Diversity Prompts [15.920623515602038]
  This paper proposes GANPrompt, a multi-dimensional large language model prompt diversity framework based on Generative Adversarial Networks (GANs) GANPrompt first trains a generator capable of producing diverse prompts by analysing multidimensional user behavioural data. These diverse prompts are then used to train the LLM to improve its performance in the face of unseen prompts.
  arXiv  Detail & Related papers  (2024-08-19T03:13:20Z)
• Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
  We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation. We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
  arXiv  Detail & Related papers  (2024-02-28T16:35:52Z)
• Diversify Question Generation with Retrieval-Augmented Style Transfer [68.00794669873196]
  We propose RAST, a framework for Retrieval-Augmented Style Transfer. The objective is to utilize the style of diverse templates for question generation. We develop a novel Reinforcement Learning (RL) based approach that maximizes a weighted combination of diversity reward and consistency reward.
  arXiv  Detail & Related papers  (2023-10-23T02:27:31Z)
• Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generation [0.0]
  We introduce Diverse Diffusion, a method for boosting image diversity beyond gender and ethnicity. Our approach contributes to the creation of more inclusive and representative AI-generated art.
  arXiv  Detail & Related papers  (2023-10-19T08:48:23Z)
• Does Writing with Language Models Reduce Content Diversity? [16.22006159795341]
  Large language models (LLMs) have led to a surge in collaborative writing with model assistance. As different users incorporate suggestions from the same model, there is a risk of decreased diversity in the produced content. We develop a set of diversity metrics and find that writing with InstructGPT (but not the GPT3) results in a statistically significant reduction in diversity.
  arXiv  Detail & Related papers  (2023-09-11T02:16:47Z)
• ComPtr: Towards Diverse Bi-source Dense Prediction Tasks via A Simple yet General Complementary Transformer [91.43066633305662]
  We propose a novel underlineComPlementary underlinetransformer, textbfComPtr, for diverse bi-source dense prediction tasks. ComPtr treats different inputs equally and builds an efficient dense interaction model in the form of sequence-to-sequence on top of the transformer.
  arXiv  Detail & Related papers  (2023-07-23T15:17:45Z)
• A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process [53.86223883060367]
  We show that diversity and coverage in unsupervised option discovery can indeed be unified under the same mathematical framework. Our proposed algorithm, ODPP, has undergone extensive evaluation on challenging tasks created with Mujoco and Atari.
  arXiv  Detail & Related papers  (2022-12-01T01:40:03Z)
• Neural Text Generation with Part-of-Speech Guided Softmax [82.63394952538292]
  We propose using linguistic annotation, i.e., part-of-speech (POS), to guide the text generation. We show that our proposed methods can generate more diverse text while maintaining comparable quality.
  arXiv  Detail & Related papers  (2021-05-08T08:53:16Z)
• Evaluating the Evaluation of Diversity in Natural Language Generation [43.05127848086264]
  We propose a framework for evaluating diversity metrics in natural language generation systems. Our framework can advance the understanding of different diversity metrics, an essential step on the road towards better NLG systems.
  arXiv  Detail & Related papers  (2020-04-06T20:44:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.