Fisher Decorator: Refining Flow Policy via A Local Transport Map

• URL: http://arxiv.org/abs/2604.17919v1
• Date: Mon, 20 Apr 2026 07:54:36 GMT
• Title: Fisher Decorator: Refining Flow Policy via A Local Transport Map
• Abstract summary: Flow-based offline reinforcement learning (RL) has achieved strong performance by parameterizing policies via flow matching.<n>Existing flow policies interpret the $L$ regularization as an upper bound of the 2-Wasserstein distance ($W$), which can be problematic in offline settings.<n>This issue stems from a fundamental geometric mismatch: the behavioral policy manifold is inherently anisotropic, whereas the $L$ regularization is isotropic and density-insensitive.<n>By analyzing the induced density transformation, we derive a local quadratic approximation of the KL-constrained objective governed by the Fisher information matrix.
• Score: 22.885775277923106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in flow-based offline reinforcement learning (RL) have achieved strong performance by parameterizing policies via flow matching. However, they still face critical trade-offs among expressiveness, optimality, and efficiency. In particular, existing flow policies interpret the $L_2$ regularization as an upper bound of the 2-Wasserstein distance ($W_2$), which can be problematic in offline settings. This issue stems from a fundamental geometric mismatch: the behavioral policy manifold is inherently anisotropic, whereas the $L_2$ (or upper bound of $W_2$) regularization is isotropic and density-insensitive, leading to systematically misaligned optimization directions. To address this, we revisit offline RL from a geometric perspective and show that policy refinement can be formulated as a local transport map: an initial flow policy augmented by a residual displacement. By analyzing the induced density transformation, we derive a local quadratic approximation of the KL-constrained objective governed by the Fisher information matrix, enabling a tractable anisotropic optimization formulation. By leveraging the score function embedded in the flow velocity, we obtain a corresponding quadratic constraint for efficient optimization. Our results reveal that the optimality gap in prior methods arises from their isotropic approximation. In contrast, our framework achieves a controllable approximation error within a provable neighborhood of the optimal solution. Extensive experiments demonstrate state-of-the-art performance across diverse offline RL benchmarks. See project page: https://github.com/ARC0127/Fisher-Decorator.

Related papers

• Learning Multimodal One-step Flow Policy via Value-weighted Optimal Transport [6.222269578057353]
  We introduce One-step Flow policy via Transport Optimal (OptiFlow), a framework for one-step flow policy learning.<n>We show that OptiFlow effectively captures optimal multimodal behaviors and achieves strong performance across diverse offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-09-14T17:10:19Z)
• Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback [14.912614425471915]
  Decision-focused learning (DFL) trains predictive models by optimizing downstream decision quality rather than standalone prediction accuracy.<n>For contextual linear optimization, most existing DFL methods assume offline full observations of objective vector.<n>We introduce a two-component hybrid estimator to update a conditional distributional model.
  arXiv  Detail & Related papers  (2026-05-31T07:58:28Z)
• Neural Network Pruning via QUBO Optimization [0.08796261172196741]
  We propose a unified Hybrid QUBO framework that bridges the importance of estimation with global optimization.<n>Our formulation integrates gradient-aware sensitivity metrics into the linear term, while utilizing data-driven similarity in the quadratic term.<n> Experiments on the SIDD image denoising dataset demonstrate that the proposed Hybrid QUBO significantly outperforms both greedy Taylor pruning and traditional L1-based QUBO.
  arXiv  Detail & Related papers  (2026-04-07T13:16:40Z)
• Variational Entropic Optimal Transport [67.76725267984578]
  We propose Variational Entropic Optimal Transport (VarEOT) for domain translation problems.<n>VarEOT is based on an exact variational reformulation of the log-partition $log mathbbE[exp(cdot)$ as a tractable generalization over an auxiliary positive normalizer.<n> Experiments on synthetic data and unpaired image-to-image translation demonstrate competitive or improved translation quality.
  arXiv  Detail & Related papers  (2026-02-02T15:48:44Z)
• FISMO: Fisher-Structured Momentum-Orthogonalized Optimizer [30.184978506988767]
  We introduce FISMO, which incorporates anisotropic neuralotropic geometry information through Fisher information geometry.<n> FISMO achieves superior efficiency and final performance compared to established baselines.
  arXiv  Detail & Related papers  (2026-01-29T14:05:04Z)
• Normalizing flows as approximations of optimal transport maps via linear-control neural ODEs [49.1574468325115]
  We consider the problem of recovering the $Wimat$-optimal transport map T between absolutely continuous measures $mu,nuinmathcalP(mathbbRn)$ as the flow of a linear-control neural ODE.
  arXiv  Detail & Related papers  (2023-11-02T17:17:03Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• Proximal Point Imitation Learning [48.50107891696562]
  We develop new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning. We leverage classical tools from optimization, in particular, the proximal-point method (PPM) and dual smoothing. We achieve convincing empirical performance for both linear and neural network function approximation.
  arXiv  Detail & Related papers  (2022-09-22T12:40:21Z)
• On the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator [12.255864026960403]
  We show how the dLQR cost varies with the coordinate transformation of the dynamic controller and then derive the optimal transformation for a given observable stabilizing controller. These results shed light on designing efficient algorithms for general decision-making problems with partially observed information.
  arXiv  Detail & Related papers  (2022-09-12T06:43:35Z)
• Optimistic Policy Optimization is Provably Efficient in Non-stationary MDPs [113.8752163061151]
  We study episodic reinforcement learning (RL) in non-stationary linear kernel Markov decision processes (MDPs)<n>We propose the underlineperiodically underlinerestarted underlineoptimistic underlinepolicy underlineoptimization algorithm (PROPO)<n>PROPO features two mechanisms: sliding-window-based policy evaluation and periodic-restart-based policy improvement.
  arXiv  Detail & Related papers  (2021-10-18T02:33:20Z)
• Iterative Amortized Policy Optimization [147.63129234446197]
  Policy networks are a central feature of deep reinforcement learning (RL) algorithms for continuous control. From the variational inference perspective, policy networks are a form of textitamortized optimization, optimizing network parameters rather than the policy distributions directly. We demonstrate that iterative amortized policy optimization, yields performance improvements over direct amortization on benchmark continuous control tasks.
  arXiv  Detail & Related papers  (2020-10-20T23:25:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.