An Interpretable Evaluation of Entropy-based Novelty of Generative Models

• URL: http://arxiv.org/abs/2402.17287v2
• Date: Fri, 14 Jun 2024 01:37:57 GMT
• Title: An Interpretable Evaluation of Entropy-based Novelty of Generative Models
• Authors: Jingwei Zhang, Cheuk Ting Li, Farzan Farnia,
• Abstract summary: We propose a Kernel-based Entropic Novelty (KEN) score to quantify the mode-based novelty of generative models. We present numerical results on synthetic and real image datasets, indicating the framework's effectiveness in detecting novel modes.
• Score: 36.29214321258605
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The massive developments of generative model frameworks require principled methods for the evaluation of a model's novelty compared to a reference dataset. While the literature has extensively studied the evaluation of the quality, diversity, and generalizability of generative models, the assessment of a model's novelty compared to a reference model has not been adequately explored in the machine learning community. In this work, we focus on the novelty assessment for multi-modal distributions and attempt to address the following differential clustering task: Given samples of a generative model $P_\mathcal{G}$ and a reference model $P_\mathrm{ref}$, how can we discover the sample types expressed by $P_\mathcal{G}$ more frequently than in $P_\mathrm{ref}$? We introduce a spectral approach to the differential clustering task and propose the Kernel-based Entropic Novelty (KEN) score to quantify the mode-based novelty of $P_\mathcal{G}$ with respect to $P_\mathrm{ref}$. We analyze the KEN score for mixture distributions with well-separable components and develop a kernel-based method to compute the KEN score from empirical data. We support the KEN framework by presenting numerical results on synthetic and real image datasets, indicating the framework's effectiveness in detecting novel modes and comparing generative models. The paper's code is available at: www.github.com/buyeah1109/KEN

Related papers

• Supervised Score-Based Modeling by Gradient Boosting [49.556736252628745]
  We propose a Supervised Score-based Model (SSM) which can be viewed as a gradient boosting algorithm combining score matching. We provide a theoretical analysis of learning and sampling for SSM to balance inference time and prediction accuracy. Our model outperforms existing models in both accuracy and inference time.
  arXiv  Detail & Related papers  (2024-11-02T07:06:53Z)
• Hierarchical Blockmodelling for Knowledge Graphs [0.5530212768657544]
  We use blockmodels for the purpose of hierarchical entity clustering on knowledge graphs. The integration of the Nested Chinese Restaurant Process and the Stick Breaking Process into the generative model allows for the induction of hierarchical clusterings. We evaluate our model on synthetic and real-world datasets and quantitatively compare against benchmark models.
  arXiv  Detail & Related papers  (2024-08-28T09:04:15Z)
• Towards a Scalable Reference-Free Evaluation of Generative Models [9.322073391374039]
  We propose a Kernel Entropy Approximation (FKEA) method to estimate VENDI and RKE entropy scores. We extensively evaluate FKEA's numerical performance in application to standard image, text, and video datasets. Our empirical results indicate the method's scalability and interpretability applied to large-scale generative models.
  arXiv  Detail & Related papers  (2024-07-03T09:54:58Z)
• Identification of Novel Modes in Generative Models via Fourier-based Differential Clustering [33.22153760327227]
  We propose a method called Fourier-based Identification of Novel Clusters (FINC) to identify modes produced by a generative model with a higher frequency. We demonstrate the application of FINC to large-scale computer vision datasets and generative model frameworks.
  arXiv  Detail & Related papers  (2024-05-04T16:06:50Z)
• Revisiting the Evaluation of Image Synthesis with GANs [55.72247435112475]
  This study presents an empirical investigation into the evaluation of synthesis performance, with generative adversarial networks (GANs) as a representative of generative models. In particular, we make in-depth analyses of various factors, including how to represent a data point in the representation space, how to calculate a fair distance using selected samples, and how many instances to use from each set.
  arXiv  Detail & Related papers  (2023-04-04T17:54:32Z)
• Sparsely constrained neural networks for model discovery of PDEs [0.0]
  We present a modular framework that determines the sparsity pattern of a deep-learning based surrogate using any sparse regression technique. We show how a different network architecture and sparsity estimator improve model discovery accuracy and convergence on several benchmark examples.
  arXiv  Detail & Related papers  (2020-11-09T11:02:40Z)
• Generative Temporal Difference Learning for Infinite-Horizon Prediction [101.59882753763888]
  We introduce the $gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. We discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors.
  arXiv  Detail & Related papers  (2020-10-27T17:54:12Z)
• Improving the Reconstruction of Disentangled Representation Learners via Multi-Stage Modeling [54.94763543386523]
  Current autoencoder-based disentangled representation learning methods achieve disentanglement by penalizing the ( aggregate) posterior to encourage statistical independence of the latent factors. We present a novel multi-stage modeling approach where the disentangled factors are first learned using a penalty-based disentangled representation learning method. Then, the low-quality reconstruction is improved with another deep generative model that is trained to model the missing correlated latent variables.
  arXiv  Detail & Related papers  (2020-10-25T18:51:15Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.