Improving Few-Step Language Flows with Untied Self-Conditioning

• URL: http://arxiv.org/abs/2608.22244v1
• Date: Sun, 23 Aug 2026 06:46:17 GMT
• Title: Improving Few-Step Language Flows with Untied Self-Conditioning
• Abstract summary: Flow-matching language models refine all token positions in parallel and can trade sampling steps for latency.<n>We trace a source of this degradation to a train--inference mismatch in previous-prediction self-conditioning.<n>We show that the mismatch degrades both the self-conditioning input and the solver update, and derive a correction for each from the model's own structure.
• Score: 17.95725125716772
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flow-matching language models refine all token positions in parallel and can trade sampling steps for latency, yet generation quality still degrades sharply with few sampling steps. We trace a source of this degradation to a train--inference mismatch in previous-prediction self-conditioning: during training, the self-conditioning input is computed from the current noisy state with no intervening solver step; during sampling, the solver folds the previous prediction into the latent before that same prediction reappears as the explicit self-conditioning input. This coupling, absent during training, creates redundancy that grows with step width. We show that the mismatch degrades both the self-conditioning input and the solver update, and derive a correction for each from the model's own structure. From the frozen projection weights we identify directions along which the self-conditioning input is redundant with the latent and dampen them; from the solver's integration structure we derive that a step-average prediction is needed and approximate it from prediction history, with scale set by offline trajectory statistics. The resulting sampler, Untied Self-Conditioning, requires no retraining and uses one evaluation per step. At 8 sampling steps on LangFlow, it reduces OpenWebText generative perplexity from $531$ to~$62$ ($8.6\times$); under an adapted Arena-Hard-Auto~v2 protocol, its outputs are preferred in $96\%$ of pairwise comparisons. On ELF-B it reduces generative perplexity from $71$ to~$43$. Improvements hold from 8 to 256 sampling steps.

Related papers

• REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation [64.84437332310516]
  We introduce textbfREER-PT, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data.<n>REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation.<n>This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training.
  arXiv  Detail & Related papers  (2026-08-31T11:34:19Z)
• FlowBender: Feedback-Aware Training for Self-Correcting Conditional Flows [18.746098727212544]
  FlowBender is a new diffusion and flow model for 3D reconstruction.<n>We show that FlowBender consistently outperforms standard supervised alignment training.<n>We propose several variants of FlowBender for differentiable operators.
  arXiv  Detail & Related papers  (2026-06-18T15:56:07Z)
• LESS Is More: Mutual-Stability Sampling for Diffusion Language Models [23.94639050546374]
  Diffusion large language models (dLLMs) offer a promising alternative to autoregressive decoding.<n>We present textscLESS, a training-free, model-agnostic adaptive sampler that treats token commitment as an online stopping problem.<n>textscLESS improves average accuracy over strong training-free adaptive samplers while using $72.1%$ fewer reverse steps than fixed-budget decoding.
  arXiv  Detail & Related papers  (2026-06-15T16:15:45Z)
• AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation [70.80207977907908]
  We introduce AnyFlow, the first any-step video diffusion distillation framework based on flow maps.<n>We show that AnyFlow achieves performance matches or surpasses consistency-based counterparts in the few-step regime.
  arXiv  Detail & Related papers  (2026-05-13T16:06:34Z)
• FlowS: One-Step Motion Prediction via Local Transport Conditioning [3.883772843505009]
  Local transport conditioning makes one-step generative motion prediction practical for safety-critical autonomy.<n>FlowS achieves state-of-the-art Soft mAP (0.4804) and mAP (0.4703) with ensemble at 75, calibrated with single-step inference.
  arXiv  Detail & Related papers  (2026-04-28T19:06:06Z)
• EFlow: Fast Few-Step Video Generator Training from Scratch via Efficient Solution Flow [52.73538517981873]
  EFlow is an efficient few-step training framework for video diffusion transformers.<n> Gated Local-Global Attention is a token-droppable hybrid block.<n>Path-Drop Guided training replaces the expensive target with a computationally cheap, weak path.
  arXiv  Detail & Related papers  (2026-03-28T02:06:55Z)
• FSampler: Training Free Acceleration of Diffusion Sampling via Epsilon Extrapolation [0.0]
  FSampler is a training free, sampler execution layer that accelerates diffusion sampling by reducing the number of function evaluations (NFE)<n>FSampler maintains a short history of denoising signals from recent real model calls and extrapolates the next epsilon using finite difference predictors.<n> operating at the sampler level, FSampler integrates with Euler/DDIM, DPM++ 2M/2S, LMS/AB2, and RES family exponential multistep methods.
  arXiv  Detail & Related papers  (2025-11-12T10:21:25Z)
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production [55.76222360698305]
  We explore a class of supervised training objectives that allow a language model to dynamically and autonomously scale the number of compute steps used for each input token.<n>For any token, the model can request additional compute steps by emitting a don't know> output.<n>We find that the CYB model requests additional steps when doing so improves accuracy, and the model adapts its processing time to token-level complexity and context.
  arXiv  Detail & Related papers  (2025-10-13T21:07:05Z)
• Self-Refining Diffusion Samplers: Enabling Parallelization via Parareal Iterations [53.180374639531145]
  Self-Refining Diffusion Samplers (SRDS) retain sample quality and can improve latency at the cost of additional parallel compute.<n>We take inspiration from the Parareal algorithm, a popular numerical method for parallel-in-time integration of differential equations.
  arXiv  Detail & Related papers  (2024-12-11T11:08:09Z)
• An Efficient Rehearsal Scheme for Catastrophic Forgetting Mitigation during Multi-stage Fine-tuning [55.467047686093025]
  A common approach to alleviate such forgetting is to rehearse samples from prior tasks during fine-tuning.<n>We propose a sampling scheme, textttbf mix-cd, that prioritizes rehearsal of collateral damage'' samples.<n>Our approach is computationally efficient, easy to implement, and outperforms several leading continual learning methods in compute-constrained settings.
  arXiv  Detail & Related papers  (2024-02-12T22:32:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.