When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation

• URL: http://arxiv.org/abs/2605.15557v1
• Date: Fri, 15 May 2026 02:53:49 GMT
• Title: When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation
• Abstract summary: This report studies a draft-conditioned latent refinement model built from a frozen BERT encoder, a parallel decoder, and a denoising DraftPrior.<n>On ROCStories, using the first two sentences as prompt and the last three as target, full 768-dimensional BERT latents recover tokens much better than compressed 256-dimensional latents.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous diffusion and flow models are attractive for non-autoregressive text generation because they can update all positions in parallel. A major difficulty is the interface between continuous latent states and discrete tokens. This report studies a draft-conditioned latent refinement model built from a frozen BERT encoder, a parallel decoder, a denoising DraftPrior, a local FlowNet, and a learned diagonal MetricNet. Early Gaussian-start experiments showed that good latent-space metrics, such as scale matching or cosine similarity, do not guarantee good decoding. Generated latents can be close to real encoder latents but still produce high-entropy, biased, or repetitive token distributions. We therefore frame the task as controlled local refinement rather than full generation from noise. On ROCStories, using the first two sentences as prompt and the last three as target, full 768-dimensional BERT latents recover tokens much better than compressed 256-dimensional latents. With 768-dimensional latents, DraftPrior target-token probability is 0.938 for clean drafts, 0.613 for 3% token dropout, 0.483 for 5% dropout, and 0.272 for 10% dropout. Local flow refinement and fused decoder-aware readout give modest additional gains, while metric learning and OT-style alignment improve geometry but do not close the decoder gap. The main result is a diagnostic one: latent geometry alone is not enough. Continuous latent text generation should be evaluated by decoder recoverability, the quality of the start distribution, and whether refinement preserves decoder-readable structure.

Related papers

• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning [61.67411833252235]
  We introduce GradCuit, which inserts optimizable latent states at a selected Transformer layer between the hidden representations of the prompt and the generated continuation.<n>Across five instruction-tuned backbones, three reasoning benchmarks, and two answer formats, GradCuit achieves an average accuracy of 64.5%.<n>For interpretability, token-level gradient attribution reveals that latent influence concentrates on reasoning-connector tokens, while layer analysis identifies early-to-middle Transformer layers as the most effective optimization space.
  arXiv  Detail & Related papers  (2026-08-03T17:55:24Z)
• Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression? [0.0]
  We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs.<n>A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace.<n>These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression.
  arXiv  Detail & Related papers  (2026-07-19T22:19:17Z)
• Why Do Few-Step Text Latents Fail When Image Latents Work? Non-Commitment at Sharp Categorical Readouts [1.1760370831826417]
  We show that a smooth, regularity-limited deterministic map cannot resolve a discrete branch choice before a sharp categorical readout.<n>In the overlapping regime of real text autoencoders, we prove that the posterior-mean terminal step flips tokens at the rate of the latent mass in an $O(s(t)$ tube around decision boundaries.
  arXiv  Detail & Related papers  (2026-06-29T14:20:57Z)
• CrossFlow: One-Step Generation Across Latent and Pixel Spaces [61.08358796319721]
  CrossFlow is a cross-space flow formulation that maps noisy latent inputs directly to pixel-space images.<n>On class-conditional ImageNet-1k at $256times256$, CrossFlow-XL achieves 1.62 FID with one function evaluation.
  arXiv  Detail & Related papers  (2026-06-18T09:10:28Z)
• Continuous Language Diffusion as a Decoder-Interface Problem [9.603960937964183]
  Continuous diffusion language models can generate fluent text from Gaussian-corrupted sentence embeddings.<n>We study this puzzle through Embedded Language Flows (ELF) and identify a decoder-basin mechanism.<n>We introduce a diagnostic protocol for denoisability, semantic recoverability, order sensitivity, decoder compatibility, and trajectory reliability.
  arXiv  Detail & Related papers  (2026-06-07T20:00:33Z)
• LaTER: Efficient Test-Time Reasoning via Latent Exploration and Explicit Verification [35.08680804423239]
  Chain-of-thought (CoT) reasoning improves large language models (LLMs) on difficult tasks, but it also makes inference expensive.<n>We propose Latent-Then-Explicit Reasoning (LaTER), a two-stage paradigm that first performs bounded exploration in a continuous latent space.<n>LaTER projects final-layer hidden states back to the input embedding space, preserves the latent KV cache, and uses entropy and model-native stop-token probes to decide when to switch.
  arXiv  Detail & Related papers  (2026-05-08T06:23:58Z)
• Coupling Models for One-Step Discrete Generation [50.37847093567075]
  Coupling Models is a one-step discrete generative model that learns a direct coupling between discrete sequences and Gaussian latents.<n>It avoids complex continuous flows over the simplex and hand-specified data-to-noise couplings.<n>It reduces LM1B text-generation perplexity by 33% at its lowest-perplexity operating point.
  arXiv  Detail & Related papers  (2026-05-08T03:40:39Z)
• Geometry-Aware Decoding with Wasserstein-Regularized Truncation and Mass Penalties for Large Language Models [9.059725329168435]
  Top-W is a geometry-aware truncation rule that uses Wasserstein distance-defined over token-embedding geometry.<n>We show that Top-W consistently outperforms prior state-of-the-art decoding approaches achieving up to 33.7% improvement.
  arXiv  Detail & Related papers  (2026-02-10T22:36:48Z)
• Continuous Autoregressive Language Models [56.49239051750678]
  We introduce Continuous Autoregressive Language Models (CALM)<n>CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector.<n>We develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling.
  arXiv  Detail & Related papers  (2025-10-31T17:58:11Z)
• Latent Refinement Decoding: Enhancing Diffusion-Based Language Models by Refining Belief States [28.663951525871756]
  We introduce Latent Refinement Decoding (LRD), a two-stage framework with Latent Refinement and a Predictive Feedback Loop.<n>LRD improves accuracy while delivering speedups of up to 10.6x, making it a strong and versatile alternative for parallel sequence generation.
  arXiv  Detail & Related papers  (2025-10-13T06:38:13Z)
• Continuous Speculative Decoding for Autoregressive Image Generation [27.308442169466975]
  Continuous visual autoregressive (AR) models have demonstrated promising performance in image generation.<n> speculative decoding has effectively accelerated discrete autoregressive inference.<n>This work addresses challenges from low acceptance rate, inconsistent output distribution, and modified distribution without analytic expression.
  arXiv  Detail & Related papers  (2024-11-18T09:19:15Z)
• Complexity Matters: Rethinking the Latent Space for Generative Modeling [65.64763873078114]
  In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity.
  arXiv  Detail & Related papers  (2023-07-17T07:12:29Z)
• Look-back Decoding for Open-Ended Text Generation [62.53302138266465]
  We propose Look-back, an improved decoding algorithm that tracks the distribution distance between current and historical decoding steps. Look-back can automatically predict potential repetitive phrase and topic drift, and remove tokens that may cause the failure modes. We perform decoding experiments on document continuation and story generation, and demonstrate that Look-back is able to generate more fluent and coherent text.
  arXiv  Detail & Related papers  (2023-05-22T20:42:37Z)
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
  We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse. We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
  arXiv  Detail & Related papers  (2020-04-30T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.