Probing the Latent Hierarchical Structure of Data via Diffusion Models

• URL: http://arxiv.org/abs/2410.13770v1
• Date: Thu, 17 Oct 2024 17:08:39 GMT
• Title: Probing the Latent Hierarchical Structure of Data via Diffusion Models
• Authors: Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi, Matthieu Wyart,
• Abstract summary: We show that experiments in diffusion-based models are a promising tool to probe the latent structure of data. We confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data.
• Score: 47.56642214162824
• License:
• Abstract: High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward-backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, are a promising tool to probe the latent structure of data. We predict in simple hierarchical models that, in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data using diffusion models.

Related papers

• Scaling Laws with Hidden Structure [2.474908349649168]
  Recent advances suggest that text and image data contain such hidden structures, which help mitigate the curse of dimensionality. In this paper, we present a controlled experimental framework to test whether neural networks can indeed exploit such hidden factorial structures'' We find that they do leverage these latent patterns to learn discrete distributions more efficiently, and derive scaling laws linking model sizes, hidden factorizations, and accuracy.
  arXiv  Detail & Related papers  (2024-11-02T22:32:53Z)
• Latent diffusion models for parameterization and data assimilation of facies-based geomodels [0.0]
  Diffusion models are trained to generate new geological realizations from input fields characterized by random noise. Latent diffusion models are shown to provide realizations that are visually consistent with samples from geomodeling software.
  arXiv  Detail & Related papers  (2024-06-21T01:32:03Z)
• Distribution-Aware Data Expansion with Diffusion Models [55.979857976023695]
  We propose DistDiff, a training-free data expansion framework based on the distribution-aware diffusion model. DistDiff consistently enhances accuracy across a diverse range of datasets compared to models trained solely on original data.
  arXiv  Detail & Related papers  (2024-03-11T14:07:53Z)
• A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data [55.748186000425996]
  Recent advancements show that diffusion models can generate high-quality images. We study this phenomenon in a hierarchical generative model of data. Our analysis characterises the relationship between time and scale in diffusion models.
  arXiv  Detail & Related papers  (2024-02-26T19:52:33Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Fascinating Supervisory Signals and Where to Find Them: Deep Anomaly Detection with Scale Learning [11.245813423781415]
  We devise novel data-driven supervision for data by introducing a characteristic -- scale -- as data labels. Scales serve as labels attached to transformed representations, thus offering ample labeled data for neural network training. This paper further proposes a scale learning-based anomaly detection method.
  arXiv  Detail & Related papers  (2023-05-25T14:48:00Z)
• Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data [68.62134204367668]
  This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. The generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution.
  arXiv  Detail & Related papers  (2023-02-14T17:02:35Z)
• Data from Model: Extracting Data from Non-robust and Robust Models [83.60161052867534]
  This work explores the reverse process of generating data from a model, attempting to reveal the relationship between the data and the model. We repeat the process of Data to Model (DtM) and Data from Model (DfM) in sequence and explore the loss of feature mapping information. Our results show that the accuracy drop is limited even after multiple sequences of DtM and DfM, especially for robust models.
  arXiv  Detail & Related papers  (2020-07-13T05:27:48Z)
• Variational Autoencoder with Learned Latent Structure [4.41370484305827]
  We introduce the Variational Autoencoder with Learned Latent Structure (VAELLS) VAELLS incorporates a learnable manifold model into the latent space of a VAE. We validate our model on examples with known latent structure and also demonstrate its capabilities on a real-world dataset.
  arXiv  Detail & Related papers  (2020-06-18T14:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.