Latent-Kernel Discrete Flow Maps for Few-Step Generation

• URL: http://arxiv.org/abs/2607.27529v1
• Date: Wed, 29 Jul 2026 23:46:31 GMT
• Title: Latent-Kernel Discrete Flow Maps for Few-Step Generation
• Abstract summary: We build a flow-map kernel that is a mixture of M factorized components tied by a single latent.<n>We show that a single step places mass on correlated completions with the same sampling time complexity as a factorized model.<n>Experiments for unconditional text generation on the One-Billion-Word (LM1B) and WikiText-103 benchmarks show that our LKF model learns strongly heterogeneous components.
• Score: 4.159429302112398
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete diffusion and flow-matching models denoise a sequence over many steps, but to keep each step cheap, they factorize the transition across positions and decide every token independently. This makes few-step generation challenging for text when the target couples two positions, such as a subject and a verb that must agree. An independent update commits to them separately, and many function evaluations are spent repairing the mismatch. Existing few-step methods buy back the lost correlation by distilling or rectifying a slow teacher, and so inherit the teacher's quality ceiling. We ask instead whether a model can express correlated steps natively, and answer with Latent-Kernel Discrete Flow Maps (LKF), a from-scratch flow-map kernel that is a mixture of M factorized components tied by a single shared latent. Conditioned on the latent, each component is cheap, and the mixture is summed over the latent in closed form for small M. We show that a single step places mass on correlated completions with the same sampling time complexity as a factorized model, since one latent is drawn per sequence and reused across the entire denoising trajectory. We also show that the Masked Diffusion Language Model (MDLM) is a special case of our LKF model at M=1. The experiments for unconditional text generation on the One-Billion-Word (LM1B) and WikiText-103 benchmarks show that our LKF model learns strongly heterogeneous components and improves generative perplexity by 2.1x to 3.3x over the likelihood baselines without losing diversity. The gain grows with M, and at M=8, it surpasses distilled and rectified few-step samplers. The source code is available at: https://github.com/mansoor181/lkf.git

Related papers

• What Iterated Self-Feeding Probes of Language Models Measure, and a test that separates the construction from the model [0.0]
  A class of methods probes a language model by feeding it its own output.<n>We ask what such a probe measures, in a construction chosen to make the question sharp.<n>The answer is that it measures two different things at once, in readings that look alike.
  arXiv  Detail & Related papers  (2026-08-11T14:40:35Z)
• UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective [17.48959208263029]
  We propose a simple continual pre-training approach for adapting pretrained GPT2 checkpoints to uniform-noise diffusion.<n>ours steadily improves the trade-off between generative perplexity (GenPPL) and unigram entropy as the sampling budget increases.<n>At both scales, ours also achieves the highest WinoGrande, SIQA, and BBH accuracy among the compared diffusion models.
  arXiv  Detail & Related papers  (2026-07-27T14:40:26Z)
• Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design [59.05127237532803]
  Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.<n>We study this sensitivity through a principled analysis of generalization error and identify three critical factors.<n>We propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling.
  arXiv  Detail & Related papers  (2026-06-13T14:41:55Z)
• Scaling Categorical Flow Maps [29.25832277355597]
  Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling.<n>We show how to generate diverse, high-quality text in as few as $4$ inference steps while maintaining near-data-level token entropy.<n>We uncover some of the challenges that arise from training these models at scale, and we provide prescriptive insights on loss weighting and time scheduling.
  arXiv  Detail & Related papers  (2026-05-08T14:50:57Z)
• Coupling Models for One-Step Discrete Generation [50.37847093567075]
  Coupling Models is a one-step discrete generative model that learns a direct coupling between discrete sequences and Gaussian latents.<n>It avoids complex continuous flows over the simplex and hand-specified data-to-noise couplings.<n>It reduces LM1B text-generation perplexity by 33% at its lowest-perplexity operating point.
  arXiv  Detail & Related papers  (2026-05-08T03:40:39Z)
• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• Adversarial Schrödinger Bridge Matching [66.39774923893103]
  Iterative Markovian Fitting (IMF) procedure alternates between Markovian and reciprocal projections of continuous-time processes. We propose a novel Discrete-time IMF (D-IMF) procedure in which learning of processes is replaced by learning just a few transition probabilities in discrete time. We show that our D-IMF procedure can provide the same quality of unpaired domain translation as the IMF, using only several generation steps instead of hundreds.
  arXiv  Detail & Related papers  (2024-05-23T11:29:33Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.