Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy

• URL: http://arxiv.org/abs/2607.21372v1
• Date: Thu, 23 Jul 2026 14:37:30 GMT
• Title: Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy
• Abstract summary: We introduce emphmean-to-score (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map.<n>For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly.
• Score: 17.48959208263029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Score Entropy Discrete Diffusion (SEDD) parameterizes discrete reverse processes with unconstrained positive score ratios. While positivity guarantees nonnegative reverse jump rates, it does not ensure Bayes realizability: ratios at a noisy state need not be jointly induced by any clean-token posterior under the forward kernel. The score-entropy loss has the correct population optimum but does not enforce this constraint away from it. In a trained pure-uniform SEDD checkpoint, roughly one quarter of complete score vectors violate the coordinate box, while more than half lie inside it yet remain materially incompatible with any valid posterior. Such violations can produce negative pre-normalization weights in finite-step sampling. Projecting raw scores onto the bridge polytope removes all observed negative weights and improves external generative PPL from $203.6$ to $175.1$ without changing the sampler. We introduce \emph{mean-to-score} (M2S), which predicts a clean-token posterior mean and converts it to the score through an exact kernel-dependent linear map. The construction applies to any known coordinate-wise continuous-time Markov chain (CTMC) satisfying a mild support condition. For uniform corruption, it maps the probability simplex onto the bridge polytope; for absorbing-mask corruption, the resulting objective recovers MD4 exactly. In a controlled 28.4M-parameter CIFAR-10 comparison, M2S lowers test BPD from $3.173$ to $3.129$ and FID-50k from $\CifarSEDDFID$ to $\CifarMtwoSFID$. A 170M-parameter M2S model trained on about 262B OpenWebText token slots outperforms the evaluated pure-uniform SEDD, GIDD, and Neural CTMC checkpoints at every tested sampling budget, reaching generative PPL $143.3$ at 128 steps versus $183.6$ for the strongest pure-uniform baseline.

Related papers

• Lévy Attention: Single-Pass Predictive Uncertainty for Continuous-Time Attention [0.0]
  Lévy Attention is a cross-attention operator whose output is integral against an inhomogeneous random measure.<n>On t-PatchGNN operator swap costs at most 5.6% accuracy against a matched control and nothing on the sparsest dataset.<n>One pass ranks 3,383 unseen patients by trust in 1.4 seconds.
  arXiv  Detail & Related papers  (2026-08-19T17:50:16Z)
• Detecting and Discriminating Operator Misspecification in Hybrid PDE-Parameter Learning: a Reference-Free Instrument, with Discrimination Bounded In Sample [0.0]
  We build an instrument that reads from a single fit and with no oracle.<n>We report where the instrument is blind, a pre-registered negative where a neural estimator loses to Tikhonov-regularized inversion at recovery.
  arXiv  Detail & Related papers  (2026-08-04T21:20:44Z)
• One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence [41.99844472131922]
  A single human must audit $N$ LLM agents under a budget of $B ll N$ audits per round.<n>We model this as budgeted noisy inspection over a two-level Gaussian copula and locate the miscalibration threshold $*$ past which confidence-ranked auditing is emphworse than random.
  arXiv  Detail & Related papers  (2026-07-30T14:52:36Z)
• Calibrated e-CUSUM Decoding for Quantized Reasoning Models: Why Token Log-Probability Is the Wrong Observable for Decoding Monitors [0.6999740786886536]
  Low-bit quantization makes small reasoning models inexpensive to deploy but can degrade their chains of thought.<n>We show that the centered token log-probability increment $log p(w_t)+H_t$ is the wrong observable for this purpose.<n>We introduce a training-free decoding controller that combines a degeneration-aware alarm score fusing token uncertainty with explicit repetition.
  arXiv  Detail & Related papers  (2026-07-13T09:34:36Z)
• ESPO: Early-Stopping Proximal Policy Optimization [78.79610718910628]
  ESPO (Early-Stopping Proximal Policy Optimization) detects trajectory failure on-the-fly and terminates rollouts early.<n>On DeepSeek-R1-Distill-Qwen-7B trained for mathematical reasoning, ESPO surpasses PPO on AIME2024 (46.28% vs. 45.25%), AMC2023 (85.83% vs. 82.94%), and MATH-500 (87.42% vs. 85.43%)
  arXiv  Detail & Related papers  (2026-05-28T12:40:22Z)
• Hardware Validation of DAGI via a Modular "Ridge" Signature and High-Order Synergistic Information [0.0]
  DAGI (Directed Acyclic Graph Information) framework on IBM Quantum hardware.<n>Small, controlled experiment whose ideal output distribution is constrained to a low-dimensional modular manifold (a "ridge)<n>Key recovery exceeds chance: per-shot accuracy 0.1689 (chance 0.125, 95% Wilson CI [0.1610, 0.1772])<n>These results support the claim that DAGI detects and quantifies non-trivial, hardware-resilient information structure.
  arXiv  Detail & Related papers  (2026-04-16T14:16:59Z)
• Online Covariance Estimation in Averaged SGD: Improved Batch-Mean Rates and Minimax Optimality via Trajectory Regression [12.805268849262243]
  We study online covariance matrix estimation for Polyak-Ruppert averaged gradient descent (SGD)<n>The construction reveals that the bottleneck is the sublinear accumulation of information about the Hessian from the SGD drift.
  arXiv  Detail & Related papers  (2026-04-12T20:49:33Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Spectral Sentinel: Scalable Byzantine-Robust Decentralized Federated Learning via Sketched Random Matrix Theory on Blockchain [0.0]
  Byzantine clients poison gradients under heterogeneous (Non-IID) data.<n>We propose Spectral Sentinel, a Byzantine detection and aggregation framework.<n>We implement the full system with blockchain integration on Polygon networks.
  arXiv  Detail & Related papers  (2025-12-14T09:43:03Z)
• TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG [46.122203287541005]
  Training-free Adaptive Retrieval Gating (TARG) is a single-shot policy that decides when to retrieve using only a short, no-latency draft from the base model.<n>On NQ-Open, TriviaQA, and PopQA, TARG consistently shifts the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2025-11-12T23:09:52Z)
• Variance-Aware Feel-Good Thompson Sampling for Contextual Bandits [54.220839560203096]
  We present FGTSVA, a variance-aware Thompson Sampling algorithm for contextual bandits with general reward function with optimal regret bound.<n>With the new decoupling coefficient denoted by $mathrmdc$, FGTS-VA achieves the regret of $tildeO(sqrtmathrmdccdotlog|mathcalF|$, where $|mathcalF|$ is the size of the model space.<n>In the setting of contextual linear bandits, the regret bound of FGTSVA matches that of UCB-based
  arXiv  Detail & Related papers  (2025-11-03T23:25:41Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Your Absorbing Discrete Diffusion Secretly Models the Bayesian Posterior [0.0]
  We prove that the expected denoiser output under the forward corruption distribution recovers the true posterior.<n>We introduce an inference-time ensemble that runs K independent denoising passes and aggregates both posterior means and variances.
  arXiv  Detail & Related papers  (2025-07-10T09:42:47Z)
• Beyond likelihood ratio bias: Nested multi-time-scale stochastic approximation for likelihood-free parameter estimation [49.78792404811239]
  We study inference in simulation-based models where the analytical form of the likelihood is unknown.<n>We use a ratio-free nested multi-time-scale approximation (SA) method that simultaneously tracks the score and drives the parameter update.<n>We show that our algorithm can eliminate the original bias $Obig(sqrtfrac1Nbig)$ and accelerate the convergence rate from $Obig(beta_k+sqrtfracalpha_kNbig)$.
  arXiv  Detail & Related papers  (2024-11-20T02:46:15Z)
• Near Sample-Optimal Reduction-based Policy Learning for Average Reward MDP [58.13930707612128]
  This work considers the sample complexity of obtaining an $varepsilon$-optimal policy in an average reward Markov Decision Process (AMDP) We prove an upper bound of $widetilde O(H varepsilon-3 ln frac1delta)$ samples per state-action pair, where $H := sp(h*)$ is the span of bias of any optimal policy, $varepsilon$ is the accuracy and $delta$ is the failure probability.
  arXiv  Detail & Related papers  (2022-12-01T15:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.