Conditional Vendi Score: An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models

• URL: http://arxiv.org/abs/2411.02817v1
• Date: Tue, 05 Nov 2024 05:30:39 GMT
• Title: Conditional Vendi Score: An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models
• Authors: Mohammad Jalali, Azim Ospanov, Amin Gohari, Farzan Farnia,
• Abstract summary: We introduce the Conditional-Vendi score based on $H(X|T)$ to quantify the internal diversity of the model. We conduct several numerical experiments to show the correlation between the Conditional-Vendi score and the internal diversity of text-conditioned generative models.
• Score: 15.40817940713399
• License:
• Abstract: Text-conditioned generation models are commonly evaluated based on the quality of the generated data and its alignment with the input text prompt. On the other hand, several applications of prompt-based generative models require sufficient diversity in the generated data to ensure the models' capability of generating image and video samples possessing a variety of features. However, most existing diversity metrics are designed for unconditional generative models, and thus cannot distinguish the diversity arising from variations in text prompts and that contributed by the generative model itself. In this work, our goal is to quantify the prompt-induced and model-induced diversity in samples generated by prompt-based models. We propose an information-theoretic approach for internal diversity quantification, where we decompose the kernel-based entropy $H(X)$ of the generated data $X$ into the sum of the conditional entropy $H(X|T)$, given text variable $T$, and the mutual information $I(X; T)$ between the text and data variables. We introduce the \emph{Conditional-Vendi} score based on $H(X|T)$ to quantify the internal diversity of the model and the \emph{Information-Vendi} score based on $I(X; T)$ to measure the statistical relevance between the generated data and text prompts. We provide theoretical results to statistically interpret these scores and relate them to the unconditional Vendi score. We conduct several numerical experiments to show the correlation between the Conditional-Vendi score and the internal diversity of text-conditioned generative models. The codebase is available at \href{https://github.com/mjalali/conditional-vendi}{https://github.com/mjalali/conditional-vendi}.

Related papers

• GRADE: Quantifying Sample Diversity in Text-to-Image Models [66.12068246962762]
  We propose GRADE: Granular Attribute Diversity Evaluation, an automatic method for quantifying sample diversity. We measure the overall diversity of 12 T2I models using 400 concept-attribute pairs, revealing that all models display limited variation. Our work proposes a modern, semantically-driven approach to measure sample diversity and highlights the stunning homogeneity in outputs by T2I models.
  arXiv  Detail & Related papers  (2024-10-29T23:10:28Z)
• An Online Learning Approach to Prompt-based Selection of Generative Models [23.91197677628145]
  An online identification of the best generation model for various input prompts can reduce the costs associated with querying sub-optimal models. We propose an online learning framework to predict the best data generation model for a given input prompt. Our experiments on real and simulated text-to-image and image-to-text generative models show RFF-UCB performs successfully in identifying the best generation model.
  arXiv  Detail & Related papers  (2024-10-17T07:33:35Z)
• An Interpretable Evaluation of Entropy-based Novelty of Generative Models [36.29214321258605]
  We propose a Kernel-based Entropic Novelty (KEN) score to quantify the mode-based novelty of generative models. We present numerical results on synthetic and real image datasets, indicating the framework's effectiveness in detecting novel modes.
  arXiv  Detail & Related papers  (2024-02-27T08:00:52Z)
• JPAVE: A Generation and Classification-based Model for Joint Product Attribute Prediction and Value Extraction [59.94977231327573]
  We propose a multi-task learning model with value generation/classification and attribute prediction called JPAVE. Two variants of our model are designed for open-world and closed-world scenarios. Experimental results on a public dataset demonstrate the superiority of our model compared with strong baselines.
  arXiv  Detail & Related papers  (2023-11-07T18:36:16Z)
• Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias [92.41919689753051]
  Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. We investigate training data generation with diversely attributed prompts, which have the potential to yield diverse and attributed generated data. We show that attributed prompts outperform simple class-conditional prompts in terms of the resulting model's performance.
  arXiv  Detail & Related papers  (2023-06-28T03:31:31Z)
• Distribution Aware Metrics for Conditional Natural Language Generation [3.6350564275444173]
  We argue that existing metrics are not appropriate for domains such as visual description or summarization where ground truths are semantically diverse. We propose a novel paradigm for multi-candidate evaluation of conditional language generation models.
  arXiv  Detail & Related papers  (2022-09-15T17:58:13Z)
• Generating Diverse Descriptions from Semantic Graphs [38.28044884015192]
  We present a graph-to-text model, incorporating a latent variable in an an-decoder model, and its use in an ensemble. We show an ensemble of models produces diverse sets of generated sentences, while retaining similar quality to state-of-the-art models. We evaluate the models on WebNLG datasets in English and Russian, and show an ensemble of models produces diverse sets of generated sentences, while retaining similar quality to state-of-the-art models.
  arXiv  Detail & Related papers  (2021-08-12T11:00:09Z)
• On the Generative Utility of Cyclic Conditionals [103.1624347008042]
  We study whether and how can we model a joint distribution $p(x,z)$ using two conditional models $p(x|z)$ that form a cycle. We propose the CyGen framework for cyclic-conditional generative modeling, including methods to enforce compatibility and use the determined distribution to fit and generate data.
  arXiv  Detail & Related papers  (2021-06-30T10:23:45Z)
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models [95.8037674226622]
  We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
  arXiv  Detail & Related papers  (2021-02-17T18:25:30Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.