Taming Sampling Perturbations with Variance Expansion Loss for Latent Diffusion Models

• URL: http://arxiv.org/abs/2603.21085v1
• Date: Sun, 22 Mar 2026 06:56:12 GMT
• Title: Taming Sampling Perturbations with Variance Expansion Loss for Latent Diffusion Models
• Abstract summary: Latent diffusion models have emerged as the dominant framework for high-fidelity and efficient image generation.<n>We show that robustness to sampling perturbations plays a crucial role in determining generation quality.<n>We propose a solution that constructs a latent space robust to sampling perturbations while maintaining strong reconstruction.
• Score: 22.168463445392856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Latent diffusion models have emerged as the dominant framework for high-fidelity and efficient image generation, owing to their ability to learn diffusion processes in compact latent spaces. However, while previous research has focused primarily on reconstruction accuracy and semantic alignment of the latent space, we observe that another critical factor, robustness to sampling perturbations, also plays a crucial role in determining generation quality. Through empirical and theoretical analyses, we show that the commonly used $β$-VAE-based tokenizers in latent diffusion models, tend to produce overly compact latent manifolds that are highly sensitive to stochastic perturbations during diffusion sampling, leading to visual degradation. To address this issue, we propose a simple yet effective solution that constructs a latent space robust to sampling perturbations while maintaining strong reconstruction fidelity. This is achieved by introducing a Variance Expansion loss that counteracts variance collapse and leverages the adversarial interplay between reconstruction and variance expansion to achieve an adaptive balance that preserves reconstruction accuracy while improving robustness to stochastic sampling. Extensive experiments demonstrate that our approach consistently enhances generation quality across different latent diffusion architectures, confirming that robustness in latent space is a key missing ingredient for stable and faithful diffusion sampling.

Related papers

• Diffusing in the Right Space: A Systematic Study of Latent Diffusability [11.450517054285788]
  Latent diffusion models leverage visual tokenizers to compress images into latent spaces for efficient generative modeling.<n>We conduct a systematic study of latent diffusability by training a large collection of tokenizers with diverse regularization strategies.<n>We identify several latent properties that consistently correlate with generation quality and exhibit strong generalization across experimental settings.
  arXiv  Detail & Related papers  (2026-06-02T12:47:14Z)
• Latent Diffusion for Missing Data [6.869298033298736]
  We investigate whether shifting diffusion to a learned latent representation improves robustness under missing-completely-at-random (MCAR) corruption.<n>For downstream imputation, latent diffusion also achieves consistently better performance than pixel-space diffusion.
  arXiv  Detail & Related papers  (2026-05-27T12:57:09Z)
• Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine [60.669081685261965]
  Diffusion models generate high-dimensional data with remarkable quality.<n>How their training efficiently learns the score function remains theoretically unexplained.<n>We instantiate this principle as Score-induced Latent Diffusion (SiLD)
  arXiv  Detail & Related papers  (2026-05-16T16:51:10Z)
• Learning Interpretable PDE Representations for Generative Reconstructions with Structured Sparsity [0.0]
  We introduce LatentPDE, a latent diffusion framework designed to simultaneously resolve sparse-observation reconstruction and super-resolution.<n>We parameterize the latent variables directly as the coefficients and source terms of an assumed governing PDE.
  arXiv  Detail & Related papers  (2026-04-26T20:14:17Z)
• On Stability and Robustness of Diffusion Posterior Sampling for Bayesian Inverse Problems [42.76879947185353]
  Diffusion-based solvers rely on a presumed likelihood for the observations in BIPs to guide the generation process.<n>We bridge this gap by characterizing the posterior approximation error and proving the emphstability of the diffusion-based solvers.<n>We propose a simple yet effective solution, emphrobust diffusion posterior sampling, which is provably emphrobust and compatible with existing gradient-based posterior samplers.
  arXiv  Detail & Related papers  (2026-02-02T12:47:15Z)
• Robust Posterior Diffusion-based Sampling via Adaptive Guidance Scale [39.27744518020771]
  We propose an adaptive likelihood step-size strategy to guide the diffusion process for inverse-problem formulations.<n>The resulting approach, Adaptive Posterior diffusion Sampling (AdaPS), is hyper-free and improves reconstruction quality across diverse imaging tasks.
  arXiv  Detail & Related papers  (2025-11-23T14:37:59Z)
• Boosting Fidelity for Pre-Trained-Diffusion-Based Low-Light Image Enhancement via Condition Refinement [63.54516423266521]
  Pre-Trained Diffusion-Based (PTDB) methods often sacrifice content fidelity to attain higher perceptual realism.<n>We propose a novel optimization strategy for conditioning in pre-trained diffusion models, enhancing fidelity while preserving realism and aesthetics.<n>Our approach is plug-and-play, seamlessly integrating into existing diffusion networks to provide more effective control.
  arXiv  Detail & Related papers  (2025-10-20T02:40:06Z)
• Provable Maximum Entropy Manifold Exploration via Diffusion Models [58.89696361871563]
  Exploration is critical for solving real-world decision-making problems such as scientific discovery.<n>We introduce a novel framework that casts exploration as entropy over approximate data manifold implicitly defined by a pre-trained diffusion model.<n>We develop an algorithm based on mirror descent that solves the exploration problem as sequential fine-tuning of a pre-trained diffusion model.
  arXiv  Detail & Related papers  (2025-06-18T11:59:15Z)
• A Study of Posterior Stability for Time-Series Latent Diffusion [59.41969496514184]
  We first show that posterior collapse will reduce latent diffusion to a variational autoencoder (VAE), making it less expressive. We then introduce a principled method: dependency measure, that quantifies the sensitivity of a recurrent decoder to input variables. Building on our theoretical and empirical studies, we introduce a new framework that extends latent diffusion and has a stable posterior.
  arXiv  Detail & Related papers  (2024-05-22T21:54:12Z)
• Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective [65.10019978876863]
  Diffusion-Based Purification (DBP) has emerged as an effective defense mechanism against adversarial attacks.<n>In this paper, we propose that the intrinsicity in the DBP process is the primary factor driving robustness.
  arXiv  Detail & Related papers  (2024-04-22T16:10:38Z)
• Digging into contrastive learning for robust depth estimation with diffusion models [55.62276027922499]
  We propose a novel robust depth estimation method called D4RD. It features a custom contrastive learning mode tailored for diffusion models to mitigate performance degradation in complex environments. In experiments, D4RD surpasses existing state-of-the-art solutions on synthetic corruption datasets and real-world weather conditions.
  arXiv  Detail & Related papers  (2024-04-15T14:29:47Z)
• Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models [24.5360032541275]
  Inverse problems arise in a multitude of applications, where the goal is to recover a clean signal from noisy and possibly (non)linear observations. Our key observation is that most existing inverse problem solvers lack the ability to adapt their compute power to the difficulty of the reconstruction task. We propose a novel method, $textitseverity encoding$, to estimate the degradation severity of corrupted signals in the latent space of an autoencoder.
  arXiv  Detail & Related papers  (2023-09-12T23:41:29Z)
• Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction [75.91471250967703]
  We introduce a novel sampling framework called Steerable Conditional Diffusion.<n>This framework adapts the diffusion model, concurrently with image reconstruction, based solely on the information provided by the available measurement.<n>We achieve substantial enhancements in out-of-distribution performance across diverse imaging modalities.
  arXiv  Detail & Related papers  (2023-08-28T08:47:06Z)
• Spontaneous Symmetry Breaking in Generative Diffusion Models [6.4322891559626125]
  Generative diffusion models have recently emerged as a leading approach for generating high-dimensional data. We show that the dynamics of these models exhibit a spontaneous symmetry breaking that divides the generative dynamics into two distinct phases. We propose a new way to understand the generative dynamics of diffusion models that has the potential to bring about higher performance and less biased fast-samplers.
  arXiv  Detail & Related papers  (2023-05-31T09:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.