Continuous Language Diffusion as a Decoder-Interface Problem

• URL: http://arxiv.org/abs/2606.08810v2
• Date: Mon, 15 Jun 2026 19:27:43 GMT
• Title: Continuous Language Diffusion as a Decoder-Interface Problem
• Abstract summary: Continuous diffusion language models can generate fluent text from Gaussian-corrupted sentence embeddings.<n>We study this puzzle through Embedded Language Flows (ELF) and identify a decoder-basin mechanism.<n>We introduce a diagnostic protocol for denoisability, semantic recoverability, order sensitivity, decoder compatibility, and trajectory reliability.
• Score: 9.603960937964183
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gaussian-corrupted sentence embeddings have no direct linguistic interpretation, yet continuous diffusion language models can generate fluent text from them. We study this puzzle through Embedded Language Flows (ELF) and identify a decoder-basin mechanism: our evidence suggests that denoising becomes reliable when trajectories reach regions where the native decoder can read stable tokens. We introduce a diagnostic protocol for denoisability, semantic recoverability, order sensitivity, decoder compatibility, and trajectory reliability. It exposes failures hidden by scalar metrics: low mean-squared error can discard linguistic content, low perplexity can reflect low-entropy collapse, and clean latent reconstruction can coexist with a narrow decoder basin. A decoder-margin bound explains why token recovery depends on margin and local decoder sensitivity, not latent error alone. Auditing public ELF checkpoints reveals an interface phase diagram: early predictions are weakly readable, mid-trajectory disagreement marks a competition region, and late predictions enter a high-margin decoder basin. Once inside, token realization is surprisingly simple on generated ELF states: frozen T5 (Text-to-Text Transfer Transformer) token-embedding lookup recovers $93$--$96\%$ of native decoder decisions, and a single linear readout reaches $97.9\%$ agreement at 32k samples, leaving an $\approx1.1$--$1.2$ perplexity gap in a structured residual tail. Under conservative held-out gates, a margin rule exits roughly $17$--$28\%$ earlier in denoising steps under an explicit diagnostic monitor. Boundary checks on LangFlow, BitstreamDiffusion, and the Continuous Latent Diffusion Language Model (Cola-DLM) show that the same interface questions remain meaningful when the state object and decoder change. Continuous and latent diffusion language models should therefore be evaluated as representation-decoder systems.

Related papers

• Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference [26.756607509798382]
  Dependency-Aware Revocable Decoding (DARD) is a training-free framework that separates tokens into masked, candidate, and unmasked states.<n>DARD consistently improves the speed-quality frontier over recent decoding methods.
  arXiv  Detail & Related papers  (2026-08-27T03:34:47Z)
• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens [22.687236570529844]
  We propose ASRD (Anchor Supervised Revocable Decoding), a training-free framework that operates within the embedding space.<n>We show that ASRD outperforms recent remasking baselines, achieving accuracy improvements of up to 6.4% while accelerating inference throughput by up to 7.2times$.
  arXiv  Detail & Related papers  (2026-06-15T15:23:47Z)
• Supportive Token Revealing for Fast Diffusion Language Model Decoding [10.022716564968695]
  AXON is a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models.<n>We show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.
  arXiv  Detail & Related papers  (2026-06-02T21:35:54Z)
• The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models [52.93186090124315]
  We argue that token commitment can be learned as a reusable trace-state policy.<n>We introduce TraceLock, a lightweight plug-in controller that instantiates this policy for a frozen diffusion language model.
  arXiv  Detail & Related papers  (2026-05-23T18:23:46Z)
• End-to-End Intracortical Speech Decoding from Neural Activity [0.8209843760716958]
  We propose an end-to-end Conformer-based neural decoder trained directly on intracortical recordings from a participant with ALS.<n>Without any external language model, the system achieves a character error rate (CER) of 23.80% on held-out validation data.
  arXiv  Detail & Related papers  (2026-05-23T00:39:59Z)
• When Latent Geometry Is Not Enough: Draft-Conditioned Latent Refinement for Non-Autoregressive Text Generation [0.0]
  This report studies a draft-conditioned latent refinement model built from a frozen BERT encoder, a parallel decoder, and a denoising DraftPrior.<n>On ROCStories, using the first two sentences as prompt and the last three as target, full 768-dimensional BERT latents recover tokens much better than compressed 256-dimensional latents.
  arXiv  Detail & Related papers  (2026-05-15T02:53:49Z)
• STDec: Spatio-Temporal Stability Guided Decoding for dLLMs [49.55447757907809]
  Large Diffusion Language Models (dLLMs) have achieved rapid progress, viewed as a promising alternative to the autoregressive paradigm.<n>Most dLLM decoders still adopt a global confidence threshold, and do not model local context from neighboring decoded states or temporal consistency of predicted token IDs.<n>We observe strong stability neartemporal stability in dLLM decoding, and our approach includes spatial-aware decoding and temporal-aware decoding.
  arXiv  Detail & Related papers  (2026-04-07T18:13:31Z)
• Escaping the BLEU Trap: A Signal-Grounded Framework with Decoupled Semantic Guidance for EEG-to-Text Decoding [41.209385903282374]
  SemKey is a novel framework that enforces signal-grounded generation through four semantic objectives.<n>We move beyond standard translation metrics by adopting N-way Retrieval Accuracy and Fréchet Distance.
  arXiv  Detail & Related papers  (2026-02-09T02:47:07Z)
• STEAD: Robust Provably Secure Linguistic Steganography with Diffusion Language Model [71.35577462669856]
  We propose a robust, provably secure linguistic steganography with diffusion language models (DLMs)<n>We introduce error correction strategies, including pseudo-random error correction and neighborhood search correction, during steganographic extraction.
  arXiv  Detail & Related papers  (2026-01-21T08:58:12Z)
• Languages are Modalities: Cross-Lingual Alignment via Encoder Injection [0.8461674097042394]
  We present a compute efficient language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder.<n>LLINK substantially improves bilingual retrieval and achieves 81.3% preference over the base model.<n>We find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment.
  arXiv  Detail & Related papers  (2025-10-31T07:43:21Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Provably Secure Disambiguating Neural Linguistic Steganography [66.30965740387047]
  The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures.<n>We propose a novel secure disambiguation method named SyncPool, which effectively addresses the segmentation ambiguity problem.<n> SyncPool does not change the size of the candidate pool or the distribution of tokens and thus is applicable to provably secure language steganography methods.
  arXiv  Detail & Related papers  (2024-03-26T09:25:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.