Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models

• URL: http://arxiv.org/abs/2605.27975v2
• Date: Thu, 28 May 2026 06:48:04 GMT
• Title: Continual Learning in Modern Hopfield Networks with an Application to Diffusion Models
• Abstract summary: We study continual learning in generative models using Hopfield energy.<n>We show that high-energy, outlier-like samples undergo a larger energy increase than cluster-like samples.<n>We also analyze memory replay and show that replay is particularly effective for high-energy samples.
• Score: 10.0426843232642
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generative models, including diffusion models, are increasingly used as foundation models and adapted through sequential fine-tuning, making continual learning an essential problem setting. However, continual learning in such generative models remains poorly understood: after a task change, what aspects of the learned distribution are most easily lost, and what replay samples should be prioritized? We address these questions through the modern Hopfield energy. Recent links between modern Hopfield networks (MHNs) and diffusion models allow analyses in MHNs to be transferred to diffusion models. We introduce intrinsic forgetting as an increase in Hopfield energy after the task change. In tractable settings in an MHN, we prove that high-energy, outlier-like samples undergo a larger energy increase than cluster-like samples, implying that samples located in sharp, isolated basins are more forgettable. We further analyze memory replay and show that replay is particularly effective for high-energy samples, enabling an energy-based selection of replay samples. We validate these predictions in experiments on MHNs and two diffusion models under continual-learning settings: Stable Diffusion and a pixel-space DDPM. In these diffusion models, Hopfield energy tracks reconstruction-based forgetting, and replay experiments reveal energy-dependent mitigation of forgetting that is consistent with the MHN analysis.

Related papers

• Generating Physically Consistent Molecules with Energy-Based Models [6.4105959373235875]
  We introduce EBMol, an energy-based model that restores inductive bias by learning an atom-additive scalar potential without explicit simulation.<n>EBMol is the first EBM for 3D molecular generation to achieve state-of-the-art performance on QM9 and GEOM-Drugs.
  arXiv  Detail & Related papers  (2026-05-18T13:26:52Z)
• Consistent Sampling and Simulation: Molecular Dynamics with Energy-Based Diffusion Models [50.77646970127369]
  We propose an energy-based diffusion model with a Fokker--Planck-derived regularization term to enforce consistency.<n>We demonstrate our approach by sampling and simulating multiple biomolecular systems, including fast-folding proteins.
  arXiv  Detail & Related papers  (2025-06-20T16:38:29Z)
• Progressive Inference-Time Annealing of Diffusion Models for Sampling from Boltzmann Densities [93.13866975467549]
  We propose Progressive Inference-Time Annealing (PITA) to learn diffusion-based samplers.<n>PITA combines two complementary techniques: Annealing of the Boltzmann distribution and Diffusion smoothing.<n>It enables equilibrium sampling of N-body particle systems, Alanine Dipeptide, and tripeptides in Cartesian coordinates.
  arXiv  Detail & Related papers  (2025-06-19T17:14:22Z)
• Memorization to Generalization: Emergence of Diffusion Models from Associative Memory [27.677278702224687]
  We study the memorization-generalization phenomenon in diffusion models via the lens of AMs.<n>Our findings provide: a novel perspective on the memorization-generalization phenomenon in diffusion models via the lens of AMs.
  arXiv  Detail & Related papers  (2025-05-27T21:20:57Z)
• Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models [12.327318533784961]
  We present a maximum reinforcement learning (IRL) approach for improving the sample quality of diffusion generative models. We train (or fine-tune) a diffusion model using the log density estimated from training data. Our empirical studies show that diffusion models fine-tuned using DxMI can generate high-quality samples in as few as 4 and 10 steps.
  arXiv  Detail & Related papers  (2024-06-30T08:52:17Z)
• An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimization [59.63880337156392]
  Diffusion models have achieved tremendous success in computer vision, audio, reinforcement learning, and computational biology. Despite the significant empirical success, theory of diffusion models is very limited. This paper provides a well-rounded theoretical exposure for stimulating forward-looking theories and methods of diffusion models.
  arXiv  Detail & Related papers  (2024-04-11T14:07:25Z)
• Diffusion Models in Vision: A Survey [73.10116197883303]
  A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage.<n> Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens.
  arXiv  Detail & Related papers  (2022-09-10T22:00:30Z)
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models [76.76860707897413]
  Current best practice advocates for a large T to ensure that the forward dynamics brings the diffusion sufficiently close to a known and simple noise distribution. We show how an auxiliary model can be used to bridge the gap between the ideal and the simulated forward dynamics, followed by a standard reverse diffusion process.
  arXiv  Detail & Related papers  (2022-06-10T15:09:46Z)
• Particle Dynamics for Learning EBMs [83.59335980576637]
  Energy-based modeling is a promising approach to unsupervised learning, which yields many downstream applications from a single model. The main difficulty in learning energy-based models with the "contrastive approaches" is the generation of samples from the current energy function at each iteration. This paper proposes an alternative approach to getting these samples and avoiding crude MCMC sampling from the current model.
  arXiv  Detail & Related papers  (2021-11-26T23:41:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.