Diffusion Models and Concept Formation

• URL: http://arxiv.org/abs/2609.13047v1
• Date: Fri, 11 Sep 2026 16:43:48 GMT
• Title: Diffusion Models and Concept Formation
• Abstract summary: We argue that diffusion models implicitly perform the same computation as Cobweb.<n>We find that Cobweb learns a discrete tree incrementally, whereas a diffusion model encodes a continuous, interpolable hierarchy in a single learned score field.<n>This reframes diffusion as a cognitive model of concept formation and offers Cobweb a continuous, scalable instantiation.
• Score: 2.9491967009603255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans organize knowledge into a taxonomy of concepts with nested levels of abstraction and a \emph{basic level} at which people recognize and name objects with the least cognitive effort. Cobweb is a classic cognitive account of this ability, an incremental learner that builds a probabilistic concept hierarchy by maximizing category utility. We argue that diffusion models, although designed for image synthesis, implicitly perform the same computation. The noisy marginals of a diffusion model are Gaussian smoothings of the data distribution, and the modes of these marginals form a hierarchy that corresponds to a Cobweb tree of probabilistic prototypes in four respects. Both are hierarchical density models, both are hierarchical-Bayesian models with Gaussian prototypes, both treat categorization as score-following that reduces uncertainty, and in both a basic level emerges. We locate this basic level for a diffusion model at an intermediate noise level, where recent analyses show that the reverse process commits to the class identity of a sample. The two models differ mainly in how they represent and learn the taxonomy. Cobweb learns a discrete tree incrementally, whereas a diffusion model encodes a continuous, interpolable hierarchy in a single learned score field fit to the data distribution. We test the correspondence on MNIST and Fashion-MNIST by recovering the diffusion hierarchy through mode-finding and comparing the basic levels of the two models. This reframes diffusion as a cognitive model of concept formation and offers Cobweb a continuous, scalable instantiation.

Related papers

• The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models [8.547042933482645]
  We study class-dependent learning in score-based diffusion models.<n>We reveal the explicit hierarchy governing these dynamics.<n>We validate our theoretical predictions empirically using U-Net models trained on Fashion MNIST.
  arXiv  Detail & Related papers  (2026-05-07T14:44:53Z)
• Discriminative protein sequence modelling with Latent Space Diffusion [0.9187159782788576]
  We explore a framework for protein sequence representation learning that decomposes the task between manifold learning and distributional modelling.<n>We present a Latent Space Diffusion architecture which combines a protein sequence autoencoder with a denoising diffusion model operating on its latent space.
  arXiv  Detail & Related papers  (2025-03-24T11:03:57Z)
• How Compositional Generalization and Creativity Improve as Diffusion Models are Trained [82.08869888944324]
  How many samples do generative models need in order to learn composition rules?<n>What signal in the data is exploited to learn those rules?<n>We discuss connections between the hierarchical clustering mechanism we introduce here and the renormalization group in physics.
  arXiv  Detail & Related papers  (2025-02-17T18:06:33Z)
• Likelihood Training of Cascaded Diffusion Models via Hierarchical Volume-preserving Maps [19.573246885611923]
  We show that cascaded models can be excellent likelihood models, so long as we overcome a fundamental difficulty with probabilistic multi-scale models.<n>Chiefly, in cascaded models each intermediary scale introduces extraneous variables that cannot be tractably marginalized out for likelihood evaluation.<n>We show that the Laplacian pyramid and wavelet transform also produces significant improvements to the state-of-the-art on a selection of benchmarks in likelihood modeling.
  arXiv  Detail & Related papers  (2025-01-13T01:20:23Z)
• A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data [51.03144354630136]
  Recent advancements show that diffusion models can generate high-quality images.<n>We study this phenomenon in a hierarchical generative model of data.<n>We find that the backward diffusion process acting after a time $t$ is governed by a phase transition.
  arXiv  Detail & Related papers  (2024-02-26T19:52:33Z)
• Training Class-Imbalanced Diffusion Model Via Overlap Optimization [55.96820607533968]
  Diffusion models trained on real-world datasets often yield inferior fidelity for tail classes. Deep generative models, including diffusion models, are biased towards classes with abundant training images. We propose a method based on contrastive learning to minimize the overlap between distributions of synthetic images for different classes.
  arXiv  Detail & Related papers  (2024-02-16T16:47:21Z)
• Guided Diffusion from Self-Supervised Diffusion Features [49.78673164423208]
  Guidance serves as a key concept in diffusion models, yet its effectiveness is often limited by the need for extra data annotation or pretraining. We propose a framework to extract guidance from, and specifically for, diffusion models.
  arXiv  Detail & Related papers  (2023-12-14T11:19:11Z)
• Hierarchically branched diffusion models leverage dataset structure for class-conditional generation [0.6800113478497425]
  Branched diffusion models rely on the same diffusion process as traditional models, but learn reverse diffusion separately for each branch of a hierarchy. We extensively evaluate branched diffusion models on several benchmark and large real-world scientific datasets spanning many data modalities.
  arXiv  Detail & Related papers  (2022-12-21T05:27:23Z)
• Neural Representations Reveal Distinct Modes of Class Fitting in Residual Convolutional Networks [5.1271832547387115]
  We leverage probabilistic models of neural representations to investigate how residual networks fit classes. We find that classes in the investigated models are not fitted in an uniform way. We show that the uncovered structure in neural representations correlate with robustness of training examples and adversarial memorization.
  arXiv  Detail & Related papers  (2022-12-01T18:55:58Z)
• OCD: Learning to Overfit with Conditional Diffusion Models [95.1828574518325]
  We present a dynamic model in which the weights are conditioned on an input sample x. We learn to match those weights that would be obtained by finetuning a base model on x and its label y.
  arXiv  Detail & Related papers  (2022-10-02T09:42:47Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• Argmax Flows and Multinomial Diffusion: Towards Non-Autoregressive Language Models [76.22217735434661]
  This paper introduces two new classes of generative models for categorical data: Argmax Flows and Multinomial Diffusion. We demonstrate that our models perform competitively on language modelling and modelling of image segmentation maps.
  arXiv  Detail & Related papers  (2021-02-10T11:04:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.