Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading

• URL: http://arxiv.org/abs/2606.00997v1
• Date: Sun, 31 May 2026 04:25:36 GMT
• Title: Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading
• Abstract summary: Order-agnostic language models (OALMs) are trained to predict masked tokens under arbitrary conditioning sets.<n>We show that learning conditionals are not exact factorizations of a coherent joint distribution.<n>We propose a complementary diagnostic based on the shape of the confidence trace.
• Score: 0.6916773850242582
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Order-agnostic language models (OALMs), including discrete diffusion language models (dLLMs), are trained to predict masked tokens under arbitrary conditioning sets, allowing sequences to be generated or scored under arbitrary reveal orders at inference time. In LLaDA-2.1, we report three findings. First, the learned conditionals are not exact factorizations of a coherent joint distribution: changing only the reveal order shifts target log-likelihood by up to 0.49 nats/token, so likelihood alone mixes content difficulty with path-dependent artifacts. Second, although confidence-first (CF) decoding is order-agnostic, its reveal orders are close to left-to-right (L2R) on content tokens. Third, we propose a complementary diagnostic based on the shape of the confidence trace. A uniform-spreading theorem shows that, at fixed total likelihood, target recoverability is maximized when per-step confidence is spread uniformly; the resulting deviation motivates $\mathrm{Var}(\log q_t)$ as a diagnostic for comparing decoding paths. Across C4 and four downstream benchmarks, low variance separates structured paths from random ordering, and variance is consistently associated with downstream correctness. These results support reporting mean confidence and confidence variance jointly when comparing OALM decoding paths.

Related papers

• When Does Consensus Mean Correctness? Measuring the Agreement-Accuracy Coupling with Semantics-Preserving Re-Rendering [0.8594140167290097]
  A model's agreement across perturbed inputs is used as a label-free reliability signal.<n>Rerendering beats resampling on both accuracy and reliability.<n>Fine-tuning on the model's own cross-render consensus inverts.
  arXiv  Detail & Related papers  (2026-08-06T07:08:05Z)
• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches [18.727291516223115]
  Desc2Fix is a framework for measuring semantic alignment between bug reports, triggering tests, and developer-written fixes.<n>We analyze 2,857 report-test-patch triplets from Defects4J and SWT-Bench using GPT-4o and DeepSeek-Chat.
  arXiv  Detail & Related papers  (2026-07-20T22:32:23Z)
• Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement [66.67463557274358]
  Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes.<n>We introduce Confidence-guided Concept Mining (CBCM) to identify reliable spurious attributes without attribute annotations.<n>We propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches.
  arXiv  Detail & Related papers  (2026-06-23T05:28:47Z)
• CauTion: Knowing When to Trust LLMs for Ensemble Causal Discovery [51.07538881798502]
  Large language models (LLMs) offer a promising source of domain knowledge to complement statistical inference.<n>We propose CauTion, a framework that reliably integrates LLM domain knowledge into an ensemble of statistical causal discovery algorithms.<n>CauTion consistently outperforms both data-centric and LLM-augmented baselines.
  arXiv  Detail & Related papers  (2026-06-02T13:07:43Z)
• Grounded Decoding: Retrieval-Anchored Probability Fusion for Faithful RAG [6.324418485299426]
  Large language models may still prioritize parametric knowledge over retrieved information when conflicts arise.<n>We propose a novel training-free decoding framework, emphGrounded Decoding, to improve factual consistency in RAG without modifying model parameters.<n>Our results indicate that probability-level fusion provides a strong and efficient alternative to logit-level intervention methods for faithful RAG decoding.
  arXiv  Detail & Related papers  (2026-05-29T23:47:39Z)
• A Mutual Information Lower Bound for Multimodal Regression Active Learning [9.04041860173466]
  Mutual Information Lower Bound (MI-LB) is a closed-form approximation for Mixture Network ensembles.<n>We show that MI-LB matches or beats every baseline evaluated on benchmarks featuring multimodal systems.
  arXiv  Detail & Related papers  (2026-05-14T14:50:47Z)
• DPRM: A Plug-in Doob h transform-induced Token-Ordering Module for Diffusion Language Models [76.12556589212666]
  We introduce DPRM (Doob h-transform Process Reward Model), a plug-in token-ordering module for diffusion language models.<n>DPRM starts from confidence-driven progressive ordering and gradually shifts to Doob h transform Process Reward guided ordering through online estimates.<n>Under tractable optimization assumptions, DPRM also yields a sample-complexity advantage over random and confidence-only ordering.
  arXiv  Detail & Related papers  (2026-04-27T11:50:26Z)
• Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models [21.699371484195865]
  We propose DEMASK, a lightweight dependency predictor that attaches to the final hidden states of a dLLM.<n> Empirically, DEMASK achieves 1.7-2.2$times$ speedup on Dream-7B while matching or improving accuracy compared to confidence-based and KL-based baselines.
  arXiv  Detail & Related papers  (2026-04-02T22:21:24Z)
• Adaptation to Intrinsic Dependence in Diffusion Language Models [5.185131234265025]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive (AR) approaches.<n>We introduce a distribution-agnostic unmasking schedule for DLMs that adapts to the (unknown) dependence structure of the target data distribution.<n>Our results significantly improve upon prior convergence theories and yield substantial sampling acceleration for low-complexity distributions.
  arXiv  Detail & Related papers  (2026-02-23T18:41:34Z)
• dgMARK: Decoding-Guided Watermarking for Diffusion Language Models [5.43345665278304]
  dgMARK is a decoding-guided watermarking method for discrete diffusion language models.<n>dgMARK steers the unmasking order toward positions whose high-reward candidate tokens satisfy a simple parity constraint.<n> Watermarks are detected via elevated parity-matching statistics.
  arXiv  Detail & Related papers  (2026-01-30T13:51:20Z)
• DiffGRM: Diffusion-based Generative Recommendation Model [63.35379395455103]
  Generative recommendation (GR) is an emerging paradigm that represents each item via a tokenizer as an n-digit semantic ID (SID)<n>We propose DiffGRM, a diffusion-based GR model that replaces the autoregressive decoder with a masked discrete diffusion model (MDM)<n> Experiments show consistent gains over strong generative and discriminative recommendation baselines on multiple datasets.
  arXiv  Detail & Related papers  (2025-10-21T03:23:32Z)
• Latent Veracity Inference for Identifying Errors in Stepwise Reasoning [78.29317733206643]
  We introduce Veracity Search (VS), a discrete search algorithm over veracity assignments.<n>It performs otherwise intractable inference in the posterior distribution over latent veracity values.<n>It generalizes VS, enabling accurate zero-shot veracity inference in novel contexts.
  arXiv  Detail & Related papers  (2025-05-17T04:16:36Z)
• Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression [102.24287051757469]
  We study self-supervised covariance estimation in deep heteroscedastic regression.<n>We derive an upper bound on the 2-Wasserstein distance between normal distributions.<n>Experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.
  arXiv  Detail & Related papers  (2025-02-14T22:37:11Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.