When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities

• URL: http://arxiv.org/abs/2608.19644v1
• Date: Thu, 20 Aug 2026 05:21:48 GMT
• Title: When Guidance Goes Off-Scale: Recalibrating Diffusion Transformers under Analog Compute-in-Memory Nonidealities
• Abstract summary: Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers.<n>We characterize the impact of analog CIM nonidealities on DiT sampling.<n>We propose a retraining-free, sampler-side recalibration that adjusts only the CFG scale for a given CIM condition.
• Score: 2.005299372367689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion Transformers (DiTs) incur high memory traffic and energy costs because sampling repeatedly evaluates large denoisers dominated by linear operations. Analog compute-in-memory (CIM) can alleviate these costs by executing linear operations within weight-storing memory arrays. However, CIM nonidealities perturb effective weights, with errors accumulating along the state-dependent denoising trajectory; their interaction with classifier-free guidance (CFG) remains underexplored. In this paper, we characterize the impact of analog CIM nonidealities on DiT sampling. Although conditional and unconditional predictions can each remain close to their clean counterparts, their difference (the CFG residual) is disproportionately attenuated and rotated. Identifying this residual as a controllable failure channel, we propose a retraining-free, sampler-side recalibration that adjusts only the CFG scale for a given CIM condition. Trajectory-level analysis shows that moderate recalibration strengthens the target-oriented component preserved in the distorted residual, enabling earlier commitment to a prompt-consistent semantic region. In contrast, excessive guidance amplifies the full noisy residual and degrades quality, resulting in a finite, noise-dependent optimum. Extensive experiments on PixArt-Sigma, PixArt-alpha, and DiT-XL/2 show that the optimal guidance scale increases with CIM noise. Using 30,000 samples per condition, guidance recalibration consistently restores generation quality across simulated CIM mappings, closing at least 87% of the CIM-induced FID gap at a CIM noise level of 0.20. It reduces FID from 59.22 to 20.49 on PixArt-Sigma, 72.37 to 21.12 on PixArt-alpha, and 20.89 to 6.62 on DiT-XL/2.

Related papers

• The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization [50.59470683896735]
  min-max optimization has attracted increasing attention due to its applications in modern machine learning.<n>We provide a comprehensive theoretical of gradient descent (SGDA)<n>We develop new clipping-free algorithms, i.e., Stoc-GDAM and Stoc-TRmax, and they both can achieve the optimal dependence on the target accuracy without using gradient clipping.
  arXiv  Detail & Related papers  (2026-09-05T12:49:28Z)
• ASSERT: Adaptive Stochastic Sampling for Robust Diffusion Models on Analog Compute-in-Memory Hardware [13.123394042691642]
  Diffusion models achieve strong image generation quality but incur high iterative denoising costs.<n>We show that the early, high-noise denoising stage is substantially more vulnerable than the final refinement stage.<n>We propose ASSERT, a training sampler that uses higher deterministicity early and smoothly denoising.
  arXiv  Detail & Related papers  (2026-09-01T09:13:10Z)
• Measured-Subspace Consistency: A Plug-and-Play Operator for Diffusion Posterior Sampling in Accelerated MRI Reconstruction [2.4827516038838064]
  Diffusion posterior samplers for accelerated MRI can reconstruct accurately yet still disagree on the acquired k-space across samples.<n>We identify this measured-subspace leakage as a physical-admissibility failure.<n>We present Measured-Subspace Consistency (MSC), a training-free terminal correction that wraps any compatible image-space posterior sampler with a standard multi-coil consistency lock.
  arXiv  Detail & Related papers  (2026-06-26T08:49:39Z)
• ALCL: An Adaptive Log-Correntropy Loss for Robust Learning under Non-Gaussian Noise [0.4078247440919472]
  We propose an Adaptive Log-Correntropy Loss (ALCL) that adaptively learns its robustness geometry during optimization.<n>ALCL consistently outperforms MSE and optimally generalized correntropy losses in both reconstruction fidelity and downstream classification accuracy.<n>These results demonstrate that adaptive robustness through joint learning of loss parameters provides a computationally efficient alternative to static correntropy-based losses for deep learning in non-Gaussian environments.
  arXiv  Detail & Related papers  (2026-06-14T22:49:57Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Tumor-aware augmentation with task-guided attention analysis improves rectal cancer segmentation from magnetic resonance images [1.3958880590777554]
  We show two interacting failure modes in CT-to-MRI transfer.<n>Inefficient token usage caused by zero-padding to match pretrained input dimensions and ineffective feature adaptation.<n>Fine-tuning on identical rectal MRI datasets improved detection rates to 224/247 (90.7%) for SMIT and 219/247 (88.7%) for Swin UNETR.
  arXiv  Detail & Related papers  (2026-05-06T23:46:46Z)
• Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory [77.27772368491698]
  Spectrals such as Muon have recently shown strong empirical performance in large-scale language model training.<n>We study this question through the linear associative memory problem.<n>We show that the storage capacity of Muon significantly exceeds that of SGD.
  arXiv  Detail & Related papers  (2026-03-27T16:13:18Z)
• Variational Rectification Inference for Learning with Noisy Labels [74.85528327499662]
  We propose variational rectification inference (VRI) to formulate the adaptive rectification for loss functions.<n>VRI is constructed as a hierarchical Bayes by treating the rectifying vector as a latent variable.<n>By introducing the variational term in VRI, the conditional posterior is estimated accurately and avoids collapsing to a Dirac delta function.
  arXiv  Detail & Related papers  (2026-03-18T01:25:08Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Introducing Fractional Classification Loss for Robust Learning with Noisy Labels [2.312414367096445]
  We introduce Fractional Classification Loss (FCL), an adaptive robust loss that automatically calibrates its robustness to label noise during training.<n>FCL achieves state-of-the-art results without the need for manual hyper parameter tuning.
  arXiv  Detail & Related papers  (2025-08-08T14:20:52Z)
• Triply Laplacian Scale Mixture Modeling for Seismic Data Noise Suppression [51.87076090814921]
  Sparsity-based tensor recovery methods have shown great potential in suppressing seismic data noise.<n>We propose a novel triply Laplacian scale mixture (TLSM) approach for seismic data noise suppression.
  arXiv  Detail & Related papers  (2025-02-20T08:28:01Z)
• Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise [60.92029979853314]
  We investigate the roles of gradient normalization and clipping in ensuring the convergence of Gradient Descent (SGD) under heavy-tailed noise. Our work provides the first theoretical evidence demonstrating the benefits of gradient normalization in SGD under heavy-tailed noise. We introduce an accelerated SGD variant incorporating gradient normalization and clipping, further enhancing convergence rates under heavy-tailed noise.
  arXiv  Detail & Related papers  (2024-10-21T22:40:42Z)
• Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion [56.38386580040991]
  Consistency Trajectory Model (CTM) is a generalization of Consistency Models (CM) CTM enables the efficient combination of adversarial training and denoising score matching loss to enhance performance. Unlike CM, CTM's access to the score function can streamline the adoption of established controllable/conditional generation methods.
  arXiv  Detail & Related papers  (2023-10-01T05:07:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.