Sampling from Flow Language Models via Marginal-Conditioned Bridges

• URL: http://arxiv.org/abs/2605.13681v1
• Date: Wed, 13 May 2026 15:38:48 GMT
• Title: Sampling from Flow Language Models via Marginal-Conditioned Bridges
• Abstract summary: Flow Language Models (FLMs) adapt continuous flow matching for one-hot encoded token sequences.<n>We argue that the natural sampler for an FLM is instead posterior-predictive.<n>We show that the induced one-step bridge kernel preserves all token-wise posterior-predictive marginals.
• Score: 1.43163477873944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flow Language Models (FLMs) are a recently introduced class of language models which adapt continuous flow matching for one-hot encoded token sequences. Their denoisers have a special structure absent from generic continuous diffusion models: each block of the denoising mean is a posterior marginal distribution over the clean token at that position. Standard DDPM-style samplers collapse these marginals to a single conditional-mean endpoint and bridge toward this simplex-valued point, which is generally not a valid one-hot sequence. We argue that the natural sampler for an FLM is instead posterior-predictive. At each reverse step, we sample a clean one-hot endpoint from the factorized posterior defined by the FLM token marginals, and then sample the next continuous state from the analytic Ornstein--Uhlenbeck bridge conditioned on that endpoint. The method is training-free, uses the same model evaluations as standard sampling, and gives a principled interface for token-level decoding controls such as temperature scaling and nucleus truncation. We show that, under exact posterior marginals, the endpoint approximation error is exactly the conditional multi-information among token positions. The induced one-step bridge kernel preserves all token-wise posterior-predictive marginals and loses only the residual cross-position dependence. Finally, we prove a Girsanov path-space comparison showing that the marginal-conditioned bridge has a no-larger denoising-error term than the frozen conditional-mean bridge, with strict improvement whenever intermediate coordinate-wise bridge observations reveal additional information about the clean token. Experiments with FLMs show that the sampler improves the quality--diversity tradeoff. Code is available at: github.com/imbirik/mcb.

Related papers

• What Does a Discrete Diffusion Model Learn? [71.03603607324338]
  discrete diffusion model learns: a denoiser, a score ratio, or a bridge plug-in predictor.<n>We start with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process.<n>Every identity is verified numerically, without approximation, on an exactly solvable model.
  arXiv  Detail & Related papers  (2026-07-06T17:56:11Z)
• Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation [72.05592785529404]
  We show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior.<n>We also introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations.
  arXiv  Detail & Related papers  (2026-05-21T17:27:19Z)
• Entropy Across the Bridge: Conditional-Marginal Discretization for Flow and Schrödinger Samplers [68.9946633641494]
  Flow matching and Schrdinger bridges define probability, yet their inference grids are usually or inherited from one-endpoint diffusion.<n>We derive a conditional-marginal entropy-rate objective for bridge-aware discretization, separating endpoint-conditioned bridge geometry from marginal flow evolution.<n>On EDM/CIFAR-10, the entropic time-discretization gives the best tested five-step FID (186.3 pm 4.0 versus 200.5 pm 2.9 for linear and 238.0 pm 5.3 for cosine)
  arXiv  Detail & Related papers  (2026-05-15T16:11:10Z)
• StippleDiffusion: Capacity-Constrained Stippling using Controlled Diffusion [41.23458880886283]
  Stipple patterns, point sets whose local density tracks a target image, are traditionally produced by per-density iteratives.<n>We present the first diffusion-based sampler that simultaneously satisfies a learned local point-distribution prior and a continuous, image-defined capacity constraint at inference.<n>A single trained checkpoint accepts arbitrary target densities at inference, generalizes to point budgets that were not seen during training, and produces stipples in time nearly independent of the output point count.
  arXiv  Detail & Related papers  (2026-05-15T10:12:42Z)
• QDSB: Quantized Diffusion Schrödinger Bridges [4.3331379059769395]
  We propose quantized diffusion Schrdinger bridges (QDSB), which compute the endpoint coupling on anchor-quantized endpoint distributions.<n>We show that the regularized optimal coupling is stable w.r.t. anchor quantization, with an error controlled by the quality of the anchor approximation.
  arXiv  Detail & Related papers  (2026-05-12T11:35:08Z)
• Sobolev Regularized MMD Gradient Flow [22.929984194375805]
  We propose a regularized variant of the gradient-regularized MMD flow.<n>The proposed regularization mitigates the difference between the mean kernel mean embeddings.<n>A key highlight of the proposed flow is that it is applicable in both sampling (from an unnormalized target distribution)
  arXiv  Detail & Related papers  (2026-05-12T09:58:19Z)
• Discrete Flow Matching: Convergence Guarantees Under Minimal Assumptions [13.856153548136826]
  Flow Matching has recently emerged as a popular class of generative models for simulating a target distribution.<n>We study two DFM models on $mathbbZ_md = 0,ldots,m-1d$, sampled through time discretization.<n>We derive non-asymptotic bounds in Kullback--Leibler divergence for the early-stopped version of the target distribution.
  arXiv  Detail & Related papers  (2026-05-09T11:06:33Z)
• Neural Continuous-Time Markov Chain: Discrete Diffusion via Decoupled Jump Timing and Direction [18.20305761007799]
  We propose textbfNeural CTMC, which parameterizes the reverse process through an emphexit rate (when to jump) and a emphjump distribution (where to jump) using two dedicated network heads.<n>We show that the evidence lower bound (ELBO) differs from a path-space KL divergence between the true and learned reverse processes by a $$-independent constant.<n>We also show that the tractable conditional surrogate preserves the gradients and minimizers of the corresponding marginal reverse-process objective under standard regularity assumptions.
  arXiv  Detail & Related papers  (2026-04-17T04:40:19Z)
• Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models [14.764619905977739]
  We introduce a sampler-centric oracle framework that replaces learned denoisers with an exact Hidden Markov Model posterior derived from a ground-truth Markov chain.<n>We show that few-step discrete diffusion samplers are not distributionally correct even under an oracle denoiser, with transition-level mismatch that vanishes only as the number of steps approaches the sequence length.
  arXiv  Detail & Related papers  (2026-02-23T09:06:13Z)
• Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts [64.34482582690927]
  We provide an efficient and principled method for sampling from a sequence of annealed, geometric-averaged, or product distributions derived from pretrained score-based models.<n>We propose Sequential Monte Carlo (SMC) resampling algorithms that leverage inference-time scaling to improve sampling quality.
  arXiv  Detail & Related papers  (2025-03-04T17:46:51Z)
• Distributional Diffusion Models with Scoring Rules [83.38210785728994]
  Diffusion models generate high-quality synthetic data.<n> generating high-quality outputs requires many discretization steps.<n>We propose to accomplish sample generation by learning the posterior em distribution of clean data samples.
  arXiv  Detail & Related papers  (2025-02-04T16:59:03Z)
• Targeted stochastic gradient Markov chain Monte Carlo for hidden Markov models with rare latent states [48.705095800341944]
  Markov chain Monte Carlo (MCMC) algorithms for hidden Markov models often rely on the forward-backward sampler. This makes them computationally slow as the length of the time series increases, motivating the development of sub-sampling-based approaches. We propose a targeted sub-sampling approach that over-samples observations corresponding to rare latent states when calculating the gradient of parameters associated with them.
  arXiv  Detail & Related papers  (2018-10-31T17:44:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.