Meta Flow Maps enable scalable reward alignment

• URL: http://arxiv.org/abs/2601.14430v1
• Date: Tue, 20 Jan 2026 19:39:56 GMT
• Title: Meta Flow Maps enable scalable reward alignment
• Abstract summary: Controlling generative models is computationally expensive.<n>This is because optimal alignment with a reward function--whether via inference-time steering or fine-tuning--requires estimating the value function.<n>We introduce Meta Flow Maps (MFMs), a framework consistency models and flow maps into extending the regime.
• Score: 22.109809498990362
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controlling generative models is computationally expensive. This is because optimal alignment with a reward function--whether via inference-time steering or fine-tuning--requires estimating the value function. This task demands access to the conditional posterior $p_{1|t}(x_1|x_t)$, the distribution of clean data $x_1$ consistent with an intermediate state $x_t$, a requirement that typically compels methods to resort to costly trajectory simulations. To address this bottleneck, we introduce Meta Flow Maps (MFMs), a framework extending consistency models and flow maps into the stochastic regime. MFMs are trained to perform stochastic one-step posterior sampling, generating arbitrarily many i.i.d. draws of clean data $x_1$ from any intermediate state. Crucially, these samples provide a differentiable reparametrization that unlocks efficient value function estimation. We leverage this capability to solve bottlenecks in both paradigms: enabling inference-time steering without inner rollouts, and facilitating unbiased, off-policy fine-tuning to general rewards. Empirically, our single-particle steered-MFM sampler outperforms a Best-of-1000 baseline on ImageNet across multiple rewards at a fraction of the compute.

Related papers

• Provably adaptive sampling with uniform and remasking discrete diffusion models [9.180350432640912]
  We study a first-order sampler based on the leave-one-out denoiser for uniform and remasking processes.<n>Our main result establishes an adaptive sampling guarantee.<n> Numerical experiments on structured synthetic distributions illustrate the predicted dimension-adaptive behavior.
  arXiv  Detail & Related papers  (2026-08-24T17:54:51Z)
• Stitched Value Model for Diffusion Alignment [76.52330634059528]
  Diffusion- or flow-based generative models must be aligned with task-specific rewards, such as prompt fidelity or aesthetic preference.<n>Here, we propose StitchVM, a model stitching framework that efficiently transfers reward models pretrained for clean images to the noisy latent regime.<n>We show that this approach yields improvements across a broad range of downstream steering and post-training methods.
  arXiv  Detail & Related papers  (2026-05-19T13:02:50Z)
• Scaling Categorical Flow Maps [29.25832277355597]
  Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling.<n>We show how to generate diverse, high-quality text in as few as $4$ inference steps while maintaining near-data-level token entropy.<n>We uncover some of the challenges that arise from training these models at scale, and we provide prescriptive insights on loss weighting and time scheduling.
  arXiv  Detail & Related papers  (2026-05-08T14:50:57Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Riemannian MeanFlow for One-Step Generation on Manifolds [54.09734511705173]
  Flow Matching enables simulation-free training of generative models.<n>MeanFlow can be extended to manifold-valued generation where velocities lie in location-dependent tangent spaces.<n>Experiments on spheres, tori, and SO(3) demonstrate competitive one-step sampling with improved quality-efficiency trade-offs and substantially reduced sampling cost.
  arXiv  Detail & Related papers  (2026-03-11T12:41:46Z)
• Value Flows [90.1510269525399]
  This paper uses modern, flexible flow-based models to estimate the full future return distributions.<n>Building upon the learned flow models, we estimate the return uncertainty of distinct states using a new flow derivative ODE.<n>Experiments on $37$ state-based and $25$ image-based benchmark tasks demonstrate that Value Flows achieves a $1.3times$ improvement on average in success rates.
  arXiv  Detail & Related papers  (2025-10-09T00:57:40Z)
• Learn to Guide Your Diffusion Model [84.82855046749657]
  We study a technique for improving quality of samples from conditional diffusion models.<n>We learn guidance weights $omega_c,(s,t)$, which are functions of the conditioning $c$, the time $t$ from which we denoise, and the time $s$ towards which we denoise.<n>We extend our framework to reward guided sampling, enabling the model to target distributions tilted by a reward function.
  arXiv  Detail & Related papers  (2025-10-01T12:21:48Z)
• Marginal Flow: a flexible and efficient framework for density estimation [6.94175385834858]
  Current density modeling approaches suffer from at least one of the following shortcomings: expensive training, slow inference, approximate likelihood, mode collapse or architectural constraints.<n>We propose a simple yet powerful framework that overcomes these limitations altogether.<n>We define our model $q_theta(x)$ through a parametric distribution $q(x|w)$ with latent parameters $w$.<n>Instead of directly optimizing the latent variables $w$, our idea is to marginalize them out by sampling $w$ from a learnable distribution $q_theta(w)$, hence the name Marginal Flow
  arXiv  Detail & Related papers  (2025-09-30T13:21:13Z)
• Solving dynamic portfolio selection problems via score-based diffusion models [2.355916155602588]
  We tackle the dynamic mean-variance portfolio selection problem in a it model-free manner, based on (generative) diffusion models.<n>We propose using data sampled from the real model $mathbb P$ with limited size to train a generative model $mathbb Q$.<n>With adaptive training and sampling methods that are tailor-made for time series data, we obtain bounds between $mathbb P$ and $mathbb Q$.
  arXiv  Detail & Related papers  (2025-07-14T04:41:49Z)
• Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts [64.34482582690927]
  We provide an efficient and principled method for sampling from a sequence of annealed, geometric-averaged, or product distributions derived from pretrained score-based models.<n>We propose Sequential Monte Carlo (SMC) resampling algorithms that leverage inference-time scaling to improve sampling quality.
  arXiv  Detail & Related papers  (2025-03-04T17:46:51Z)
• Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis [16.288866201806382]
  We develop a model-free RLHF best policy identification algorithm, called $mathsfBSAD$, without explicit reward model inference.<n>The algorithm identifies the optimal policy directly from human preference information in a backward manner.
  arXiv  Detail & Related papers  (2024-06-11T17:01:41Z)
• Transfer Q Star: Principled Decoding for LLM Alignment [105.89114186982972]
  Transfer $Q*$ estimates the optimal value function for a target reward $r$ through a baseline model. Our approach significantly reduces the sub-optimality gap observed in prior SoTA methods.
  arXiv  Detail & Related papers  (2024-05-30T21:36:12Z)
• Better Training of GFlowNets with Local Credit and Incomplete Trajectories [81.14310509871935]
  We consider the case where the energy function can be applied not just to terminal states but also to intermediate states. This is for example achieved when the energy function is additive, with terms available along the trajectory. This enables a training objective that can be applied to update parameters even with incomplete trajectories.
  arXiv  Detail & Related papers  (2023-02-03T12:19:42Z)
• Generalized Differentiable RANSAC [95.95627475224231]
  $nabla$-RANSAC is a differentiable RANSAC that allows learning the entire randomized robust estimation pipeline. $nabla$-RANSAC is superior to the state-of-the-art in terms of accuracy while running at a similar speed to its less accurate alternatives.
  arXiv  Detail & Related papers  (2022-12-26T15:13:13Z)
• An Improved Analysis of Gradient Tracking for Decentralized Machine Learning [34.144764431505486]
  We consider decentralized machine learning over a network where the training data is distributed across $n$ agents. The agent's common goal is to find a model that minimizes the average of all local loss functions. We improve the dependency on $p$ from $mathcalO(p-1)$ to $mathcalO(p-1)$ in the noiseless case.
  arXiv  Detail & Related papers  (2022-02-08T12:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.