Exploring the Design Space of Reward Backpropagation for Flow Matching

• URL: http://arxiv.org/abs/2606.11075v1
• Date: Tue, 09 Jun 2026 16:36:54 GMT
• Title: Exploring the Design Space of Reward Backpropagation for Flow Matching
• Abstract summary: FlowBP is a unified surrogate-trajectory framework that treats the backward trajectory itself as the design object.<n>We instantiate three variants: FlowBP-Sparse, FlowBP-Bridge, and FlowBP-Lagrange.<n>All three bound memory by the active-set size and limit gradient chaining to at most one Jacobian factor.
• Score: 47.80328464705813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Aligning text-to-image flow matching models with human preferences via direct reward backpropagation is sample-efficient but hampered by two well-known pathologies: activations cannot be stored across the full sampling trajectory at modern model scale, and chained Jacobian products across steps inflate the reward gradient as it travels back to early indices. Connector-based methods, such as LeapAlign, address these issues by replacing the full backward trajectory with a short pinned path, highlighting a useful decoupling between sampling and optimization. However, the quality of the resulting gradient depends on how accurately this short path approximates the full rollout, especially over long intervals. We propose FlowBP, a unified surrogate-trajectory framework that treats the backward trajectory itself as the design object. FlowBP keeps a no-gradient cached rollout for sampling, then builds a lightweight backward surrogate from cached and selectively re-forwarded velocities. This view separates four choices: the reward-model input, active set, integration weights, and bridge coupling, and recovers prior direct-gradient methods as particular settings. Within this framework, we instantiate three variants: FlowBP-Sparse uses sparse Euler reconstruction, FlowBP-Bridge adds controlled bridge coupling, and FlowBP-Lagrange raises the order of leap quadrature. All three bound memory by the active-set size and limit gradient chaining to at most one Jacobian factor. Across SD3.5-M, FLUX.1-dev, and FLUX.2-Klein-base on preference, quality, and compositional metrics, the three variants improve over direct-gradient baselines on most metrics.

Related papers

• Hierarchical Flow Matching for 3D Point Cloud Generation [3.451986888321395]
  Existing flow-based methods rely on continuous normalizing flows (CNFs) that demand expensive ODE solving and trace estimation during training.<n>We propose Hierarchical Flow Matching (HFM) that extends flow matching to bilevel structure for unconditional 3D point cloud generation.<n>The resulting straight OT paths enable efficient sampling with as few as 15 Euler steps per flow, while the structured latent space supports downstream tasks including classification.
  arXiv  Detail & Related papers  (2026-08-06T03:23:49Z)
• Coupled Routing and Configuration Optimization for Multi-Viewpoint Robotic Inspection [4.164290233214403]
  We present a unified framework that turns a set of 6-DoF inspection viewpoints into a time-optimal, collision-free route for a 9-DoF robotic system.<n>Unlike modular pipelines that fix a single inverse-kinematics (IK) configuration per viewpoint, build an all-pairs travel-time map, and then route, our method jointly optimize the visiting order and the per-viewpoint configuration in a single global search.
  arXiv  Detail & Related papers  (2026-06-17T17:14:54Z)
• Accelerating Rectified Flow Models via Trajectory-Aware Caching [35.13071059415995]
  We propose TACache, a training-free acceleration framework following a skip-then-compensate paradigm.<n>Trajectory-Aware Cache achieves up to 4.14 speedup on text-to-image generation and 2.11 speedup on text-to-video generation.
  arXiv  Detail & Related papers  (2026-05-16T03:44:58Z)
• Entropy Across the Bridge: Conditional-Marginal Discretization for Flow and Schrödinger Samplers [68.9946633641494]
  Flow matching and Schrdinger bridges define probability, yet their inference grids are usually or inherited from one-endpoint diffusion.<n>We derive a conditional-marginal entropy-rate objective for bridge-aware discretization, separating endpoint-conditioned bridge geometry from marginal flow evolution.<n>On EDM/CIFAR-10, the entropic time-discretization gives the best tested five-step FID (186.3 pm 4.0 versus 200.5 pm 2.9 for linear and 238.0 pm 5.3 for cosine)
  arXiv  Detail & Related papers  (2026-05-15T16:11:10Z)
• AnyFlow: Any-Step Video Diffusion Model with On-Policy Flow Map Distillation [70.80207977907908]
  We introduce AnyFlow, the first any-step video diffusion distillation framework based on flow maps.<n>We show that AnyFlow achieves performance matches or surpasses consistency-based counterparts in the few-step regime.
  arXiv  Detail & Related papers  (2026-05-13T16:06:34Z)
• FAVE: Flow-based Average Velocity Establishment for Sequential Recommendation [33.3202615024807]
  We propose a Flow-based Average Velocity Establishment (Fave) framework for one-step generation recommendation.<n>Fave learns a direct trajectory from an informative prior to the target distribution.<n>Experiments on three benchmarks demonstrate that Fave not only achieves state-of-the-art recommendation performance but also delivers an order-of-magnitude improvement in inference efficiency.
  arXiv  Detail & Related papers  (2026-04-06T05:17:43Z)
• FlowConsist: Make Your Flow Consistent with Real Trajectory [99.22869983378062]
  We argue that current fast-flow training paradigms suffer from two fundamental issues.<n> conditional velocities constructed from randomly paired noise-data samples introduce systematic trajectory drift.<n>We propose FlowConsist, a training framework designed to enforce trajectory consistency in fast flows.
  arXiv  Detail & Related papers  (2026-02-06T03:24:23Z)
• Flow-based Extremal Mathematical Structure Discovery [1.1087735229999816]
  FlowBoost is a closed-loop generative framework that learns to discover rare and extremal geometric structures.<n>We demonstrate the framework on four geometric optimization problems.
  arXiv  Detail & Related papers  (2026-01-25T21:41:47Z)
• Optimal Flow Matching: Learning Straight Trajectories in Just One Step [89.37027530300617]
  We develop and theoretically justify the novel textbf Optimal Flow Matching (OFM) approach. It allows recovering the straight OT displacement for the quadratic transport in just one FM step. The main idea of our approach is the employment of vector field for FM which are parameterized by convex functions.
  arXiv  Detail & Related papers  (2024-03-19T19:44:54Z)
• AccFlow: Backward Accumulation for Long-Range Optical Flow [70.4251045372285]
  This paper proposes a novel recurrent framework called AccFlow for long-range optical flow estimation. We demonstrate the superiority of backward accumulation over conventional forward accumulation. Experiments validate the effectiveness of AccFlow in handling long-range optical flow estimation.
  arXiv  Detail & Related papers  (2023-08-25T01:51:26Z)
• Deep Shells: Unsupervised Shape Correspondence with Optimal Transport [52.646396621449]
  We propose a novel unsupervised learning approach to 3D shape correspondence. We show that the proposed method significantly improves over the state-of-the-art on multiple datasets.
  arXiv  Detail & Related papers  (2020-10-28T22:24:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.