Disagree to Accelerate: Closing the Loop on Diffusion Feature Forecasts

• URL: http://arxiv.org/abs/2608.01740v1
• Date: Mon, 03 Aug 2026 06:08:30 GMT
• Title: Disagree to Accelerate: Closing the Loop on Diffusion Feature Forecasts
• Abstract summary: Training-free feature forecasting accelerates diffusion sampling by predicting features at skipped steps.<n>We show that reliability can be observed from the cache itself.<n>We introduce RACER, a training-free closed-loop controller with two responses.
• Score: 18.247977457930556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training-free feature forecasting accelerates diffusion sampling by predicting features at skipped denoising steps. Recent work has mainly focused on designing stronger forecasters. Yet forecast error varies sharply across steps, and open-loop caches trust the forecast in full at every skipped step. This fixed trust is what breaks as acceleration turns aggressive. The missing question is not only how to forecast better, but when and how much to trust a forecast. We show that reliability can be observed from the cache itself. Two forecasts agree where the feature trajectory is smooth, and they diverge where prediction turns hard. Their disagreement is a cheap runtime signal, and it costs no extra denoiser evaluation. Based on this signal, we introduce RACER, a training-free closed-loop controller with two responses. It continuously shrinks uncertain forecasts toward the last computed feature. At the riskiest steps, RACER refreshes the feature and repays the added evaluation by skipping a later scheduled one. We derive a deterministic error bound for the shrinkage and empirically evaluate its validity and tightness across acceleration regimes. At the same number of denoiser evaluations, RACER improves the strongest open-loop baseline across SD3.5-Large, FLUX.1-dev, Wan2.1-14B, and HunyuanVideo on DrawBench, VBench, and COCO. On SD3.5, we further show that RACER samples faster at equal quality. RACER generalizes across forecasting designs as well. For example, it recovers much of the quality lost on a Taylor base. These results show that reliable diffusion acceleration also depends on how forecasts are used. Code is available at https://github.com/LiZaiyuan0619/RACER

Related papers

• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• Adaptive Spectral Feature Forecasting for Diffusion Sampling Acceleration [58.19554276924402]
  We propose spectral diffusion feature forecaster (Spectrum) to enable global, long-range feature reuse with tightly controlled error.<n>We achieve up to 4.79$times$ speedup on FLUX.1 and 4.67$times$ speedup on Wan2.1-14B, while maintaining much higher sample quality compared with the baselines.
  arXiv  Detail & Related papers  (2026-03-02T08:59:11Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• Forecasting When to Forecast: Accelerating Diffusion Models with Confidence-Gated Taylor [10.102021461975244]
  Diffusion Transformers (DiTs) have demonstrated remarkable performance in visual generation tasks.<n>Recent training-free approaches exploit the redundancy of features across timesteps by caching and reusing past representations to accelerate inference.<n>TaylorSeer instead uses cached features to predict future ones via Taylor expansion.<n>We propose a novel approach to better leverage Taylor-based acceleration.
  arXiv  Detail & Related papers  (2025-08-04T09:39:31Z)
• Improving Prediction Certainty Estimation for Reliable Early Exiting via Null Space Projection [16.838728310658105]
  We propose a novel early exiting method based on the Certainty-Aware Probability (CAP) score.<n>We show that our method can achieve an average speed-up ratio of 2.19x across all tasks with negligible performance degradation.
  arXiv  Detail & Related papers  (2025-06-08T05:08:34Z)
• CoDiCast: Conditional Diffusion Model for Global Weather Prediction with Uncertainty Quantification [23.605859261510407]
  Traditional ensemble numerical weather prediction is computationally intensive.<n>Most existing machine learning-based weather prediction approaches are efficient and accurate.<n>We propose CoDiCast, a conditional diffusion model to generate accurate global weather prediction.
  arXiv  Detail & Related papers  (2024-09-09T18:18:47Z)
• Short-term wind speed forecasting model based on an attention-gated recurrent neural network and error correction strategy [0.0]
  This dissertation raises one short-term wind speed forecast pattern on the foundation of attention with an improved gated recurrent neural network (AtGRU) The proposed SSA-AtGRU-VMD-GRU model outperforms the compared models in three case studies on Woodburn, St. Thomas, and Santa Cruz.
  arXiv  Detail & Related papers  (2024-04-17T14:27:45Z)
• Scaling transformer neural networks for skillful and reliable medium-range weather forecasting [23.249955524044392]
  We introduce Stormer, a state-of-the-art performance on weather forecasting with minimal changes to the standard transformer backbone. At the core of Stormer is a randomized forecasting objective that trains the model to forecast the weather dynamics over varying time intervals. On WeatherBench 2, Stormer performs competitively at short to medium-range forecasts and outperforms current methods beyond 7 days.
  arXiv  Detail & Related papers  (2023-12-06T19:46:06Z)
• Streaming Motion Forecasting for Autonomous Driving [71.7468645504988]
  We introduce a benchmark that queries future trajectories on streaming data and we refer to it as "streaming forecasting" Our benchmark inherently captures the disappearance and re-appearance of agents, which is a safety-critical problem yet overlooked by snapshot-based benchmarks. We propose a plug-and-play meta-algorithm called "Predictive Streamer" that can adapt any snapshot-based forecaster into a streaming forecaster.
  arXiv  Detail & Related papers  (2023-10-02T17:13:16Z)
• Pedestrian Trajectory Forecasting Using Deep Ensembles Under Sensing Uncertainty [125.41260574344933]
  We consider an encoder-decoder based deep ensemble network for capturing both perception and predictive uncertainty simultaneously. Overall, deep ensembles provided more robust predictions and the consideration of upstream uncertainty further increased the estimation accuracy for the model.
  arXiv  Detail & Related papers  (2023-05-26T04:27:48Z)
• ZigZag: Universal Sampling-free Uncertainty Estimation Through Two-Step Inference [54.17205151960878]
  We introduce a sampling-free approach that is generic and easy to deploy. We produce reliable uncertainty estimates on par with state-of-the-art methods at a significantly lower computational cost.
  arXiv  Detail & Related papers  (2022-11-21T13:23:09Z)
• Learning to Predict Trustworthiness with Steep Slope Loss [69.40817968905495]
  We study the problem of predicting trustworthiness on real-world large-scale datasets. We observe that the trustworthiness predictors trained with prior-art loss functions are prone to view both correct predictions and incorrect predictions to be trustworthy. We propose a novel steep slope loss to separate the features w.r.t. correct predictions from the ones w.r.t. incorrect predictions by two slide-like curves that oppose each other.
  arXiv  Detail & Related papers  (2021-09-30T19:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.