Micro-Diffusion Compression - Binary Tree Tweedie Denoising for Online Probability Estimation

• URL: http://arxiv.org/abs/2603.08771v3
• Date: Thu, 12 Mar 2026 12:53:30 GMT
• Title: Micro-Diffusion Compression - Binary Tree Tweedie Denoising for Online Probability Estimation
• Abstract summary: We introduce a micro-diffusion denoising layer for improving probability estimates produced by adaptive statistical models.<n>Midicoth combines five fully online components: an adaptive PPM model, a long-range match model, a trie-based word model, a high-order context model, and the micro-diffusion denoiser applied as the final stage.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Midicoth, a lossless compression system that introduces a micro-diffusion denoising layer for improving probability estimates produced by adaptive statistical models. In compressors such as Prediction by Partial Matching (PPM), probability estimates are smoothed by a prior to handle sparse observations. When contexts have been seen only a few times, this prior dominates the prediction and produces distributions that are significantly flatter than the true source distribution, leading to compression inefficiency. Midicoth addresses this limitation by treating prior smoothing as a shrinkage process and applying a reverse denoising step that corrects predicted probabilities using empirical calibration statistics. To make this correction data-efficient, the method decomposes each byte prediction into a hierarchy of binary decisions along a bitwise tree. This converts a single 256-way calibration problem into a sequence of binary calibration tasks, enabling reliable estimation of correction terms from relatively small numbers of observations. The denoising process is applied in multiple successive steps, allowing each stage to refine residual prediction errors left by the previous one. The micro-diffusion layer operates as a lightweight post-blend calibration stage applied after all model predictions have been combined, allowing it to correct systematic biases in the final probability distribution. Midicoth combines five fully online components: an adaptive PPM model, a long-range match model, a trie-based word model, a high-order context model, and the micro-diffusion denoiser applied as the final stage.

Related papers

• From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime [52.459513910242485]
  We develop a general statistical framework for studying optimal noise-level allocation in diffusion training.<n>We show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels.<n>We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable.
  arXiv  Detail & Related papers  (2026-07-10T20:59:06Z)
• Entropy Minimization without Model Collapse: Mitigating Prediction Bias in Medical Imaging [2.651701169890824]
  Entropy minimization is the dominant objective for test-time adaptation, yet its failure mode, model collapse, remains poorly understood.<n>We propose Distribution Shift Bias Reduction (DSBR), a bias-correcting objective that specifically targets this failure mode.<n>We find that DSBR consistently stabilizes test-time adaptation, prevents model collapse, and matches or outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-06-01T14:48:16Z)
• Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers [6.018167862639113]
  We show that introducing and preconditioner are typically estimated from the same minibatch.<n>Even when the preconditioner estimate is unbiased, its inverse or inverse-root is generally biased because inversion is nonlinear.<n>We propose a single-batch bias-correction framework that addresses both effects.
  arXiv  Detail & Related papers  (2026-05-20T05:54:24Z)
• Perturb and Correct: Post-Hoc Ensembles using Affine Redundancy [0.0]
  We introduce Perturb-and-Correct (P&C), a method for constructing diverse predictors from a single pretrained network.<n>P&C applies random hidden layer perturbations with a least-squares correction in the subsequent affine layer, producing predictors that agree on calibration data while remaining free to disagree away from it.<n> Empirically, P&C achieves a strong ID/OOD tradeoff across MuJoCo dynamics prediction and CIFAR-10 OOD detection, matching or outperforming standard post-hoc baselines while requiring only a single pretrained model.
  arXiv  Detail & Related papers  (2026-05-02T22:48:11Z)
• Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs [13.741630476895773]
  This paper provides the first convergence analysis for training transformer-based diffusion models.<n>We consider the population Denoising Diffusion Probabilistic Model (DDPM) objective for denoising data.<n>A deeper investigation reveals that trained a mean denoising module enables the trained model to approximate the oracle Minimum Mean Squared Error (MMSE) of the injected noise in the diffusion steps.
  arXiv  Detail & Related papers  (2026-04-11T07:46:15Z)
• Controllable Probabilistic Forecasting with Stochastic Decomposition Layers [1.3995263206621]
  We introduce Decomposition Layers (SDL) for converting deterministic machine learning weather models into ensemble systems.<n>SDL applies learned perturbations at three decoder scales through latent-driven modulation, per-pixel noise, and channel scaling.<n>When applied to WXFormer via transfer learning, SDL requires less than 2% of the computational cost needed to train the baseline model.
  arXiv  Detail & Related papers  (2025-12-21T17:10:00Z)
• Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence [8.952347049759094]
  We construct prediction intervals for neural network regressors post-hoc without held-out data.<n>We train just once and locally perturb model parameters using Gauss-Newton influence.
  arXiv  Detail & Related papers  (2025-07-27T13:34:32Z)
• Normalizing Flows for Conformal Regression [0.0]
  Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. We present a general scheme to localize the intervals by training the calibration process. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity.
  arXiv  Detail & Related papers  (2024-06-05T15:04:28Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• On Calibrating Diffusion Probabilistic Models [78.75538484265292]
  diffusion probabilistic models (DPMs) have achieved promising results in diverse generative tasks. We propose a simple way for calibrating an arbitrary pretrained DPM, with which the score matching loss can be reduced and the lower bounds of model likelihood can be increased. Our calibration method is performed only once and the resulting models can be used repeatedly for sampling.
  arXiv  Detail & Related papers  (2023-02-21T14:14:40Z)
• Conformal Inference for Online Prediction with Arbitrary Distribution Shifts [1.2277343096128712]
  We consider the problem of forming prediction sets in an online setting where the distribution generating the data is allowed to vary over time. We develop a novel procedure with provably small regret over all local time intervals of a given width. We test our techniques on two real-world datasets aimed at predicting stock market volatility and COVID-19 case counts.
  arXiv  Detail & Related papers  (2022-08-17T16:51:12Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.