On the Interpolation Effect of Score Smoothing

• URL: http://arxiv.org/abs/2502.19499v1
• Date: Wed, 26 Feb 2025 19:04:01 GMT
• Title: On the Interpolation Effect of Score Smoothing
• Authors: Zhengdao Chen,
• Abstract summary: We study the interplay between score smoothing and the denoising dynamics with mathematically solvable models.<n>We demonstrate how a smoothed score function can lead to the generation of samples that interpolate among the training data within their subspace.<n>We also present evidence that learning score functions with regularized neural networks can have a similar effect on the denoising dynamics as score smoothing.
• Score: 8.883733362171034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Score-based diffusion models have achieved remarkable progress in various domains with the ability to generate new data samples that do not exist in the training set. In this work, we examine the hypothesis that their generalization ability arises from an interpolation effect caused by a smoothing of the empirical score function. Focusing on settings where the training set lies uniformly in a one-dimensional linear subspace, we study the interplay between score smoothing and the denoising dynamics with mathematically solvable models. In particular, we demonstrate how a smoothed score function can lead to the generation of samples that interpolate among the training data within their subspace while avoiding full memorization. We also present evidence that learning score functions with regularized neural networks can have a similar effect on the denoising dynamics as score smoothing.

Related papers

• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models [11.743167854433306]
  Diffusion models generate samples by estimating the score function of the target distribution at various noise levels.<n>In this work, we establish the first (nearly) dimension-free sample bounds complexity for learning these score functions.<n>A key aspect of our analysis is the use of a single function approximator to jointly estimate scores across noise levels.
  arXiv  Detail & Related papers  (2025-02-14T18:32:22Z)
• Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
  In this study, we explore employing neural networks as ensemble methods. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• Learning invariant representations of time-homogeneous stochastic dynamical systems [27.127773672738535]
  We study the problem of learning a representation of the state that faithfully captures its dynamics. This is instrumental to learning the transfer operator or the generator of the system. We show that the search for a good representation can be cast as an optimization problem over neural networks.
  arXiv  Detail & Related papers  (2023-07-19T11:32:24Z)
• Seismic Data Interpolation via Denoising Diffusion Implicit Models with Coherence-corrected Resampling [7.755439545030289]
  Deep learning models such as U-Net often underperform when the training and test missing patterns do not match. We propose a novel framework that is built upon the multi-modal diffusion models. Inference phase, we introduce the denoising diffusion implicit model to reduce the number of sampling steps. To enhance the coherence and continuity between the revealed traces and the missing traces, we propose two strategies.
  arXiv  Detail & Related papers  (2023-07-09T16:37:47Z)
• Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles [34.32021888691789]
  We develop a theory of feature-bagging in noisy least-squares ridge ensembles. We demonstrate that subsampling shifts the double-descent peak of a linear predictor. We compare the performance of a feature-subsampling ensemble to a single linear predictor.
  arXiv  Detail & Related papers  (2023-07-06T17:56:06Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• Deep Double Descent via Smooth Interpolation [2.141079906482723]
  We quantify sharpness of fit of training data by studying the loss landscape w.r.t. to the input variable locally to each training point. Our findings show that loss sharpness in the input space follows both model- and epoch-wise double descent, with worse peaks observed around noisy targets. While small interpolating models sharply fit both clean and noisy data, large interpolating models express a smooth loss landscape, in contrast to existing intuition.
  arXiv  Detail & Related papers  (2022-09-21T02:46:13Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• Diffusion-Based Representation Learning [65.55681678004038]
  We augment the denoising score matching framework to enable representation learning without any supervised signal. In contrast, the introduced diffusion-based representation learning relies on a new formulation of the denoising score matching objective. Using the same approach, we propose to learn an infinite-dimensional latent code that achieves improvements of state-of-the-art models on semi-supervised image classification.
  arXiv  Detail & Related papers  (2021-05-29T09:26:02Z)
• Score-informed Networks for Music Performance Assessment [64.12728872707446]
  Deep neural network-based methods incorporating score information into MPA models have not yet been investigated. We introduce three different models capable of score-informed performance assessment.
  arXiv  Detail & Related papers  (2020-08-01T07:46:24Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.