Restoration Score Distillation: From Corrupted Diffusion Pretraining to One-Step High-Quality Generation

• URL: http://arxiv.org/abs/2505.13377v1
• Date: Mon, 19 May 2025 17:21:03 GMT
• Title: Restoration Score Distillation: From Corrupted Diffusion Pretraining to One-Step High-Quality Generation
• Authors: Yasi Zhang, Tianyu Chen, Zhendong Wang, Ying Nian Wu, Mingyuan Zhou, Oscar Leong,
• Abstract summary: We propose textitRestoration Score Distillation (RSD), a principled generalization of Denoising Score Distillation (DSD)<n>RSD accommodates a broader range of corruption types, such as blurred, incomplete, or low-resolution images.<n>It consistently surpasses its teacher model across diverse restoration tasks on both natural and scientific datasets.
• Score: 82.39763984380625
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning generative models from corrupted data is a fundamental yet persistently challenging task across scientific disciplines, particularly when access to clean data is limited or expensive. Denoising Score Distillation (DSD) \cite{chen2025denoising} recently introduced a novel and surprisingly effective strategy that leverages score distillation to train high-fidelity generative models directly from noisy observations. Building upon this foundation, we propose \textit{Restoration Score Distillation} (RSD), a principled generalization of DSD that accommodates a broader range of corruption types, such as blurred, incomplete, or low-resolution images. RSD operates by first pretraining a teacher diffusion model solely on corrupted data and subsequently distilling it into a single-step generator that produces high-quality reconstructions. Empirically, RSD consistently surpasses its teacher model across diverse restoration tasks on both natural and scientific datasets. Moreover, beyond standard diffusion objectives, the RSD framework is compatible with several corruption-aware training techniques such as Ambient Tweedie, Ambient Diffusion, and its Fourier-space variant, enabling flexible integration with recent advances in diffusion modeling. Theoretically, we demonstrate that in a linear regime, RSD recovers the eigenspace of the clean data covariance matrix from linear measurements, thereby serving as an implicit regularizer. This interpretation recasts score distillation not only as a sampling acceleration technique but as a principled approach to enhancing generative performance in severely degraded data regimes.

Related papers

• ResPF: Residual Poisson Flow for Efficient and Physically Consistent Sparse-View CT Reconstruction [7.644299873269135]
  Sparse-view computed tomography (CT) is a practical solution to reduce radiation dose, but the resulting inverse problem poses significant challenges for accurate image reconstruction.<n>Recent advances in generative modeling, particularly Poisson Flow Generative Models (PFGM), enable high-fidelity image synthesis.<n>We propose Residual Poisson Flow (ResPF) Generative Models for efficient and accurate sparse-view CT reconstruction.
  arXiv  Detail & Related papers  (2025-06-06T01:43:35Z)
• Denoising Score Distillation: From Noisy Diffusion Pretraining to One-Step High-Quality Generation [82.39763984380625]
  We introduce denoising score distillation (DSD), a surprisingly effective and novel approach for training high-quality generative models from low-quality data.<n>DSD pretrains a diffusion model exclusively on noisy, corrupted samples and then distills it into a one-step generator capable of producing refined, clean outputs.
  arXiv  Detail & Related papers  (2025-03-10T17:44:46Z)
• The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model [2.355460994057843]
  Self-distillation (SD) has attracted attention as a simple yet powerful approach in machine learning.<n>Despite its widespread use, the mechanisms underlying its effectiveness remain unclear.
  arXiv  Detail & Related papers  (2025-01-27T17:20:48Z)
• Data-free Distillation with Degradation-prompt Diffusion for Multi-weather Image Restoration [29.731089599252954]
  We propose a novel Data-free Distillation with Degradation-prompt Diffusion framework for multi-weather Image Restoration (D4IR) It replaces GANs with pre-trained diffusion models to avoid model collapse and incorporates a degradation-aware prompt adapter. Our proposal achieves comparable performance to the model distilled with original training data, and is even superior to other mainstream unsupervised methods.
  arXiv  Detail & Related papers  (2024-09-05T12:07:17Z)
• Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection [19.099643719358692]
  We propose a simple yet effective two-stage industrial anomaly detection framework, termed as AAND. In the first anomaly amplification stage, we propose a novel Residual Anomaly Amplification (RAA) module to advance the pre-trained teacher encoder. We further employ a reverse distillation paradigm to train a student decoder, in which a novel Hard Knowledge Distillation (HKD) loss is built to better facilitate the reconstruction of normal patterns.
  arXiv  Detail & Related papers  (2024-05-03T13:00:22Z)
• Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation [61.03530321578825]
  We introduce Score identity Distillation (SiD), an innovative data-free method that distills the generative capabilities of pretrained diffusion models into a single-step generator. SiD not only facilitates an exponentially fast reduction in Fr'echet inception distance (FID) during distillation but also approaches or even exceeds the FID performance of the original teacher diffusion models.
  arXiv  Detail & Related papers  (2024-04-05T12:30:19Z)
• Reflected Diffusion Models [93.26107023470979]
  We present Reflected Diffusion Models, which reverse a reflected differential equation evolving on the support of the data. Our approach learns the score function through a generalized score matching loss and extends key components of standard diffusion models.
  arXiv  Detail & Related papers  (2023-04-10T17:54:38Z)
• Prompting to Distill: Boosting Data-Free Knowledge Distillation via Reinforced Prompt [52.6946016535059]
  Data-free knowledge distillation (DFKD) conducts knowledge distillation via eliminating the dependence of original training data. We propose a prompt-based method, termed as PromptDFD, that allows us to take advantage of learned language priors. As shown in our experiments, the proposed method substantially improves the synthesis quality and achieves considerable improvements on distillation performance.
  arXiv  Detail & Related papers  (2022-05-16T08:56:53Z)
• Churn Reduction via Distillation [54.5952282395487]
  We show an equivalence between training with distillation using the base model as the teacher and training with an explicit constraint on the predictive churn. We then show that distillation performs strongly for low churn training against a number of recent baselines.
  arXiv  Detail & Related papers  (2021-06-04T18:03:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.