TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models

• URL: http://arxiv.org/abs/2605.24292v1
• Date: Fri, 22 May 2026 23:48:25 GMT
• Title: TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models
• Abstract summary: We introduce the Tangent Upper Bound on Evidence (TUBE), a variational upper bound on log-likelihood that admits an unbiased Monte Carlo estimator.<n>Our TUBE extends across latent-variable models, including masked diffusion models (MDMs), any-order ARMs (AO-ARMs) and block variants of both.
• Score: 47.0940918286083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Log-likelihood is a standard metric for evaluating generative models. Unfortunately, in contrast to autoregressive models (ARMs), discrete diffusion models generally do not admit exact computation of this quantity. Existing evaluations, therefore, rely on the evidence lower bound (ELBO), leaving unclear how much higher the true value may be. We address this by introducing the Tangent Upper Bound on Evidence (TUBE), a variational upper bound on log-likelihood that admits an unbiased Monte Carlo estimator. Our TUBE extends across latent-variable models, including masked diffusion models (MDMs), any-order ARMs (AO-ARMs), and block variants of both. Applied to block MDMs and block AO-ARMs, TUBE reveals our key empirical finding that these models lie strictly below the exact ARM baseline, showing that ARMs still dominate in likelihood.

Related papers

• BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers [12.083218729202963]
  We introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion.<n> BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens.<n>With block size $16$ on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText.
  arXiv  Detail & Related papers  (2026-06-01T13:36:54Z)
• The Efficiency Gap in Byte Modeling [101.18202046105718]
  Two alternative paradigms have challenged the use of subword tokenization and autoregressive ordering.<n>We show that removing these structural priors incurs a significant computational cost.<n>Our findings suggest that future modality-agnostic designs must incorporate alternative structural biases to maintain viable scaling trajectories.
  arXiv  Detail & Related papers  (2026-05-13T03:03:30Z)
• Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction [13.764740803730803]
  We provide a unified view of autoregressive models (ARMs) and energy-based models (EBMs)<n>We derive the equivalence between supervised learning of ARMs and EBMs.<n>Our results provide insights into the ability of ARMs to plan ahead, despite being based on the next-token prediction paradigm.
  arXiv  Detail & Related papers  (2025-12-17T17:14:26Z)
• Score-based Membership Inference on Diffusion Models [3.742113529511043]
  Membership inference attacks (MIAs) against diffusion models have emerged as a pressing privacy concern.<n>We present a theoretical and empirical study of score-based MIAs, focusing on the predicted noise vectors that diffusion models learn to approximate.<n>We show that the expected denoiser output points toward a kernel-weighted local mean of nearby training samples, such that its norm encodes proximity to the training set and thereby reveals membership.
  arXiv  Detail & Related papers  (2025-09-29T16:28:55Z)
• Spatial Reasoning with Denoising Models [49.83744014336816]
  We introduce a framework to perform reasoning over sets of continuous variables via denoising generative models.<n>For the first time, that order of generation can successfully be predicted by the denoising network itself.<n>Using these findings, we can increase the accuracy of specific reasoning tasks from 1% to >50%.
  arXiv  Detail & Related papers  (2025-02-28T14:08:30Z)
• Large Language Diffusion Models [93.26422905620008]
  Large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs)<n>We introduce LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning paradigm.<n>Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines.
  arXiv  Detail & Related papers  (2025-02-14T08:23:51Z)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models [105.70889434492143]
  Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling.<n>We show that we can convert AR models ranging from 127M to 7B parameters into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training.<n>Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts.
  arXiv  Detail & Related papers  (2024-10-23T14:04:22Z)
• Offline Model-Based Reinforcement Learning with Anti-Exploration [0.0]
  We present Morse Model-based offline RL (MoMo), which extends the anti-exploration paradigm found in offline model-free RL. MoMo performs offline reinforcement learning using an anti-exploration bonus to counteract value overestimation. The latter outperforms prior model-based and model-free baselines on the majority of D4RL datasets tested.
  arXiv  Detail & Related papers  (2024-08-20T10:29:21Z)
• Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data [55.54827581105283]
  We show that the concrete score in absorbing diffusion can be expressed as conditional probabilities of clean data.<n>We propose a dedicated diffusion model without time-condition that characterizes the time-independent conditional probabilities.<n>Our models achieve SOTA performance among diffusion models on 5 zero-shot language modeling benchmarks.
  arXiv  Detail & Related papers  (2024-06-06T04:22:11Z)
• Induced Model Matching: Restricted Models Help Train Full-Featured Models [1.4963011898406866]
  We consider scenarios where a very accurate (often small) predictive model using restricted features is available when training a full-featured (often larger) model.<n>How can the restricted model be useful to the full model?<n>We introduce a methodology called Induced Model Matching (IMM)<n>IMM aligns the context-restricted, or induced, version of the large model with the restricted model.
  arXiv  Detail & Related papers  (2024-02-19T20:21:09Z)
• MAP's not dead yet: Uncovering true language model modes by conditioning away degeneracy [26.464639238671054]
  We argue that mixing even a tiny amount of low-entropy noise with a population text distribution can cause the data distribution's mode to become degenerate. We empirically verify that the length-conditional modes of machine translation models and language models are indeed more fluent and topical than their unconditional modes.
  arXiv  Detail & Related papers  (2023-11-15T09:38:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.