PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching

• URL: http://arxiv.org/abs/2603.18363v1
• Date: Thu, 19 Mar 2026 00:00:36 GMT
• Title: PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching
• Abstract summary: PowerFlow is a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem.<n>By targeting $$-power distributions, PowerFlow enables the directional elicitation of the dual nature of Large Language Models.
• Score: 34.44050784146993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unsupervised Reinforcement Learning from Internal Feedback (RLIF) has emerged as a promising paradigm for eliciting the latent capabilities of Large Language Models (LLMs) without external supervision. However, current methods rely on heuristic intrinsic rewards, which often lack a well-defined theoretical optimization target and are prone to degenerative biases. In this work, we introduce PowerFlow, a principled framework that reformulates unsupervised fine-tuning as a distribution matching problem. By casting GFlowNet as an amortized variational sampler for unnormalized densities, we propose a length-aware Trajectory-Balance objective that explicitly neutralizes the structural length biases inherent in autoregressive generation. By targeting $α$-power distributions, PowerFlow enables the directional elicitation of the dual nature of LLMs: sharpening the distribution ($α> 1$) to intensify logical reasoning, or flattening it ($α< 1$) to unlock expressive creativity. Extensive experiments demonstrate that PowerFlow consistently outperforms existing RLIF methods, matching or even exceeding supervised GRPO. Furthermore, by mitigating over-sharpening in aligned models, our approach achieves simultaneous gains in diversity and quality, shifting the Pareto frontier in creative tasks.

Related papers

• ChainFlow-VLA: Causal Flow Planning with Vision-Language Models [7.814547378443577]
  Current end-to-end autonomous driving systems are fundamentally limited by a mismatch between temporal causal reasoning and global trajectory consistency.<n>We propose ChainFlow-VLA, which unifies causal generation and global refinement within a unified probabilistic framework.<n> Experiments demonstrate that ChainFlow-VLA achieves robust planning in ambiguous and long-tail scenarios, achieving a state-of-the-art score of 94.85 on the NAVSIM v1 leaderboard.
  arXiv  Detail & Related papers  (2026-05-22T06:17:35Z)
• One-Step Generative Modeling via Wasserstein Gradient Flows [56.266175641614986]
  W-Flow is a framework for training a generator that transforms samples from a simple reference distribution into samples from a target data distribution in a single step.<n>We set a new state of the art for one-step ImageNet 256$times$256 generation, achieving 1.29 FID, with improved mode coverage and domain transfer.
  arXiv  Detail & Related papers  (2026-05-12T08:29:44Z)
• P-Flow: Proxy-gradient Flows for Linear Inverse Problems [28.438072266272954]
  P-Flow is a framework that stabilizes the reconstruction process by leveraging a proxy gradient to update the source point.<n>Experiments across diverse restoration tasks demonstrate that P-Flow delivers competitive performance.
  arXiv  Detail & Related papers  (2026-05-08T17:07:36Z)
• OpenVLThinkerV2: A Generalist Multimodal Reasoning Model for Multi-domain Visual Tasks [81.07598709704628]
  We present a novel RL training objective that replaces standard linear scaling with non-linear distributional matching.<n>We also present OpenVLThinkerV2, a highly robust, general-purpose multimodal model.
  arXiv  Detail & Related papers  (2026-04-09T17:59:39Z)
• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching [8.665369041430969]
  Flow Matching (FM) enables one-step generation, but integrating it into Entropy Reinforcement Learning (MaxEnt RL) is challenging.<n>We propose textbfFlow-based textbfLog-likelihood-textbfAware textbfMaximum textbfEntropy RL (textbfFLAME), a principled framework that addresses these challenges.
  arXiv  Detail & Related papers  (2026-02-02T03:54:11Z)
• Optimal Control Meets Flow Matching: A Principled Route to Multi-Subject Fidelity [35.95129874095729]
  Text-to-image (T2I) models excel on single-entity prompts but struggle with multi-subject descriptions.<n>We introduce the first theoretical framework with principled optimizable objective for steering sampling dynamics toward multi-subject fidelity.
  arXiv  Detail & Related papers  (2025-10-02T17:59:58Z)
• Unleashing Flow Policies with Distributional Critics [15.149475517073258]
  We introduce the Distributional Flow Critic (DFC), a novel critic architecture that learns the complete state-action return distribution.<n>DFC provides the expressive flow-based policy with a rich, distributional Bellman target, which offers a more stable and informative learning signal.
  arXiv  Detail & Related papers  (2025-09-27T03:51:06Z)
• A Theory of Multi-Agent Generative Flow Networks [65.53605277612444]
  We propose a theoretical framework for multi-agent generative flow networks (MA-GFlowNets)<n>MA-GFlowNets can be applied to multiple agents to generate objects collaboratively through a series of joint actions.<n>Joint Flow training is based on a local-global principle allowing to train a collection of (local) GFN as a unique (global) GFN.
  arXiv  Detail & Related papers  (2025-09-24T04:01:21Z)
• Mask in the Mirror: Implicit Sparsification [16.69937899343079]
  Continuous sparsification strategies are among the most effective methods for reducing inference costs and memory demands of neural networks.<n>A key factor in their success is the implicit $L_1$ regularization induced by jointly learning both mask and weight variables.<n>We provide a theoretical explanation for this observation by analyzing the learning dynamics, revealing that early continuous sparsification is governed by an implicit $L$ regularization.<n>We propose a method to dynamically control the strength of this implicit bias.
  arXiv  Detail & Related papers  (2024-08-19T13:14:02Z)
• Distributional Refinement Network: Distributional Forecasting via Deep Learning [0.8142555609235358]
  A key task in actuarial modelling involves modelling the distributional properties of losses. We propose a Distributional Refinement Network (DRN), which combines an inherently interpretable baseline model with a flexible neural network. DRN captures varying effects of features across all quantiles, improving predictive performance while maintaining adequate interpretability.
  arXiv  Detail & Related papers  (2024-06-03T05:14:32Z)
• Stochastic Generative Flow Networks [89.34644133901647]
  Generative Flow Networks (or GFlowNets) learn to sample complex structures through the lens of "inference as control" Existing GFlowNets can be applied only to deterministic environments, and fail in more general tasks with dynamics. This paper introduces GFlowNets, a new algorithm that extends GFlowNets to environments.
  arXiv  Detail & Related papers  (2023-02-19T03:19:40Z)
• Learning GFlowNets from partial episodes for improved convergence and stability [56.99229746004125]
  Generative flow networks (GFlowNets) are algorithms for training a sequential sampler of discrete objects under an unnormalized target density. Existing training objectives for GFlowNets are either local to states or transitions, or propagate a reward signal over an entire sampling trajectory. Inspired by the TD($lambda$) algorithm in reinforcement learning, we introduce subtrajectory balance or SubTB($lambda$), a GFlowNet training objective that can learn from partial action subsequences of varying lengths.
  arXiv  Detail & Related papers  (2022-09-26T15:44:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.