Recovering Hidden Reward in Diffusion-Based Policies

• URL: http://arxiv.org/abs/2605.00623v1
• Date: Fri, 01 May 2026 12:52:56 GMT
• Title: Recovering Hidden Reward in Diffusion-Based Policies
• Abstract summary: EnergyFlow is a framework that unifies generative action modeling with inverse reinforcement learning.<n>Under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function.<n>We show that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds.
• Score: 20.31180123599115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces EnergyFlow, a framework that unifies generative action modeling with inverse reinforcement learning by parameterizing a scalar energy function whose gradient is the denoising field. We establish that under maximum-entropy optimality, the score function learned via denoising score matching recovers the gradient of the expert's soft Q-function, enabling reward extraction without adversarial training. Formally, we prove that constraining the learned field to be conservative reduces hypothesis complexity and tightens out-of-distribution generalization bounds. We further characterize the identifiability of recovered rewards and bound how score estimation errors propagate to action preferences. Empirically, EnergyFlow achieves state-of-the-art imitation performance on various manipulation tasks while providing an effective reward signal for downstream reinforcement learning that outperforms both adversarial IRL methods and likelihood-based alternatives. These results show that the structural constraints required for valid reward extraction simultaneously serve as beneficial inductive biases for policy generalization. The code is available at https://github.com/sotaagi/EnergyFlow.

Related papers

• Energy-Structured Low-Rank Adaptation for Continual Learning [78.5390284258307]
  We propose textbfEnergy-Concentrated and textbfEnergy-Ordered textbfLow-textbfRank textbfAdaptation (E$2$-LoRA)<n>By explicitly ordering and concentrating knowledge into leading ranks, E$2$-LoRA frees capacity for subsequent tasks.
  arXiv  Detail & Related papers  (2026-05-26T12:55:02Z)
• Scaling Reasoning Efficiently via Relaxed On-Policy Distillation [49.39199261602195]
  REOPOLD (Relaxed On-Policy Distillation) is a framework that stabilizes optimization by relaxing the strict imitation constraints of standard on-policy distillation.<n>Specifically, REOPOLD selectively and leverages rewards from the teacher through mixture-based reward clipping, entropy-based token-level dynamic sampling, and a unified exploration-to-refinement training strategy.
  arXiv  Detail & Related papers  (2026-03-11T16:26:52Z)
• Alternating Reinforcement Learning with Contextual Rubric Rewards [2.1172256884504588]
  Reinforcement Learning with Rewards (RLRR) is a framework that extends conventional reinforcement learning from human feedback.<n>Existing approaches in RLRR are limited to linearly compressing vector rewards into a scalar reward with a fixed weightings.<n>This work proposes Alternating Reinforcement Learning with Rewards (ARL-RR) to overcome the limitations of reward aggregation.
  arXiv  Detail & Related papers  (2026-03-04T04:18:39Z)
• Unbiased Gradient Estimation for Event Binning via Functional Backpropagation [64.88399635309918]
  We propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation.<n>We achieve 9.4% lower EPE in self-supervised optical flow, and 5.1% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception.
  arXiv  Detail & Related papers  (2026-02-13T04:05:03Z)
• Boosting Maximum Entropy Reinforcement Learning via One-Step Flow Matching [8.665369041430969]
  Flow Matching (FM) enables one-step generation, but integrating it into Entropy Reinforcement Learning (MaxEnt RL) is challenging.<n>We propose textbfFlow-based textbfLog-likelihood-textbfAware textbfMaximum textbfEntropy RL (textbfFLAME), a principled framework that addresses these challenges.
  arXiv  Detail & Related papers  (2026-02-02T03:54:11Z)
• Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training [29.56905427210088]
  gradient-ARM is a framework that jointly optimize a rubric generator and a judge using reinforcement learning from preference feedback.<n>We show that gradient-ARM achieves state-of-the-art performance among baselines on benchmarks and significantly improves downstream policy alignment in both offline and online reinforcement learning settings.
  arXiv  Detail & Related papers  (2026-02-02T00:50:53Z)
• FlowSteer: Guiding Few-Step Image Synthesis with Authentic Trajectories [82.90132015584359]
  ReFlow has theoretical consistency with flow matching but suboptimal performance in practical scenarios.<n>We propose FlowSteer, a method unlocks the potential of ReFlow-based distillation by guiding the student along teacher's authentic generation trajectories.
  arXiv  Detail & Related papers  (2025-11-24T07:13:23Z)
• Latent Chain-of-Thought for Visual Reasoning [53.541579327424046]
  Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs)<n>We reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference.<n>We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-27T23:10:06Z)
• Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits [58.63897489864948]
  Reinforcement learning with outcome-based feedback faces a fundamental challenge.<n>How do we assign credit to the right actions?<n>This paper provides the first comprehensive analysis of this problem in online RL with general function approximation.
  arXiv  Detail & Related papers  (2025-05-26T17:44:08Z)
• Stochastic Forward-Forward Learning through Representational Dimensionality Compression [14.020257767487323]
  We propose a novel goodness function dimensionality compression that uses the effective dimensionality of fluctuating neural responses to incorporate second-order statistical structure.<n>Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing.
  arXiv  Detail & Related papers  (2025-05-22T13:19:29Z)
• ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards [3.1986315488647588]
  Quadrotor control policies can be trained with high performance using the exact gradients of the rewards.<n> Partially differentiable rewards will result in biased gradient propagation that degrades training performance.<n>We propose an approach that mitigates gradient bias while preserving the training efficiency of BPTT.
  arXiv  Detail & Related papers  (2025-01-24T14:18:22Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.