Context-weighted Discrete Flow Matching

• URL: http://arxiv.org/abs/2607.21427v1
• Date: Thu, 23 Jul 2026 15:29:49 GMT
• Title: Context-weighted Discrete Flow Matching
• Abstract summary: We propose a simple modification to the underlying continuous-time Markov chain (CTMC) that incorporates local context information.<n>Our context-weighted sampler improves generation quality with negligible computational overhead.<n>Our scaled cross-entropy loss function reweights the training signal from different tokens and reduces generative perplexity by up to 63% on OpenWebText.
• Score: 13.385997589934645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete flow matching provides a flexible framework for generative modeling on discrete structures. However, the standard factorized training objective exposes the model to targets of varying difficulty, mixing well-conditioned, predictable tokens with ambiguous, high-entropy ones. We empirically demonstrate that the uncertainty over the value of each token is closely related to the density of available context in its neighborhood. Motivated by this observation, we propose a simple modification to the underlying continuous-time Markov chain (CTMC) that incorporates local context information. Our context-weighted sampler improves generation quality with negligible computational overhead, while our scaled cross-entropy loss function reweights the training signal from different tokens and reduces generative perplexity by up to 63% on OpenWebText. Moreover, our approach matches a strong semi-autoregressive block diffusion baseline in quality while retaining the ability to perform generation in any order. These results highlight the role of local context as an important factor in discrete generative modeling and show that simple context-aware modifications can significantly improve both sampling and training efficiency.

Related papers

• Contrastive-Augmented Flow Matching for Style-Content Disentanglement [26.08049457095161]
  Contrastive Augmented Flow Matching (CAtFM) is a framework that integrates contrastive regularization into an invertible flow matching formulation to promote structured content-style representations.<n>Our experiments operate in the CLIP embedding space, with additional validation using frozen DINO and ALIGN encoders.<n>Overall, CAtFM provides a simple way to couple discriminative constraints with deterministic transport, improving disentanglement and robustness under distribution shift.
  arXiv  Detail & Related papers  (2026-07-14T06:23:52Z)
• Simplifying Flow Matching Transformations with Low-Rank Mixture Models [58.797704398690165]
  We propose using mixtures of probabilistic principal component analyzers (MPPCA) as the latent density for normalizing flows.<n>We simplify the learned flow transformation by learning a latent distribution that more closely aligns with the data distribution in terms of KL divergence.<n>MPPCA models can be fit quickly and cheaply using the expectation-maximization algorithm, making them a practical choice for initializing latent distributions even in high-dimensional generative tasks.
  arXiv  Detail & Related papers  (2026-06-29T03:03:06Z)
• MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation [65.068801413044]
  Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams.<n>We introduce MoASE, a mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture.
  arXiv  Detail & Related papers  (2026-05-18T01:52:12Z)
• Structured Coupling for Flow Matching [16.361601016656707]
  We propose a cooperative framework that augments flow matching with structured latent representation learning.<n>By introducing structured latent variables and noise into the source, SCFM jointly learns a structured prior and a continuous transport map.<n>It uses a shared time-dependent recognition network for both latent variable model variational inference and intermediate-time flow velocity estimation.
  arXiv  Detail & Related papers  (2026-05-08T12:47:37Z)
• Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models [60.25003813232697]
  We propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM) as a training-free adaptive guidance framework for Masked Diffusion Models (MDMs)<n>ARAM calibrates the guidance scale during denoising according to the Signal-to-Noise Ratio (SNR) of the distributional shift induced by retrieved context.<n>Experiments on multiple knowledge-intensive QA benchmarks show that ARAM improves overall QA performance over competitive RAG baselines.
  arXiv  Detail & Related papers  (2026-03-18T12:54:50Z)
• SHAMISA: SHAped Modeling of Implicit Structural Associations for Self-supervised No-Reference Image Quality Assessment [6.175621390241037]
  No-Reference Image Quality Assessment (NR-IQA) aims to estimate perceptual quality without access to a reference image of pristine quality.<n>We propose SHAMISA, a non-contrastive self-supervised framework that learns from unlabeled distorted images by leveraging explicitly structured supervision.
  arXiv  Detail & Related papers  (2026-03-14T00:37:26Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.