RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

• URL: http://arxiv.org/abs/2608.20208v1
• Date: Thu, 20 Aug 2026 16:07:56 GMT
• Title: RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation
• Abstract summary: offline reinforcement learning framework makes AR-NF policies practical for robotic manipulation.<n>RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective.<n>Experiments across simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance.
• Score: 36.16163751472016
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limiting their use in likelihood-based offline RL post-training. AR-NFs offer both expressive action modeling and exact likelihood evaluation, but their sequential sampling incurs substantial sampling overhead during policy optimization and deployment. We present RoMAN-Flow (Robotic Manipulation with Autoregressive Normalizing Flows), an offline reinforcement learning framework that makes AR-NF policies practical for robotic manipulation by addressing this sampling bottleneck in both stages. During policy optimization, RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective that assigns higher likelihood to high-advantage actions from the offline dataset without sampling from the autoregressive policy. For efficient deployment, it distills the optimized autoregressive policy into a one-step action generator, enabling low-latency action prediction. Experiments across multiple simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance while substantially reducing inference latency. Code is available at https://github.com/konnyaku28/RoMAN-Flow.

Related papers

• VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching [2.7429630700600893]
  We propose a scalable offline RL framework that enables dense value-guided shaping within a flow-based policy.<n> VGFM parameterizes the policy as a conditional flow-matching model in action (x-prediction) space.<n>We show that VGFM achieves strong performance across a wide range of tasks under rigorous evaluation protocols.
  arXiv  Detail & Related papers  (2026-09-13T03:50:52Z)
• FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space [15.351960740594201]
  FlowDAgger is a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space.<n>Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy.<n>We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation.
  arXiv  Detail & Related papers  (2026-07-09T19:07:33Z)
• From Attacks to Curricula: Learnability-Guided Adversarial Training for Safe Autonomous Driving [56.30087557121323]
  AlignADV is a learnability-guided closed-loop adversarial training framework.<n>It converts adversarial scenarios into resolvable and capability-aligned curricula.<n> Experiments show it reduces training steps by up to 40.6 percent.
  arXiv  Detail & Related papers  (2026-06-12T02:13:55Z)
• Improving Robotic Generalist Policies via Flow Reversal Steering [92.97477771370428]
  Generalist policies can learn a wide range of skills from diverse robot datasets.<n>In order to solve or improve on challenging new tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior.<n>We propose Flow Reversal Steering (FRS): a method that takes suboptimal but reasonable'' actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes.
  arXiv  Detail & Related papers  (2026-06-11T17:59:45Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• HiFlow: Tokenization-Free Scale-Wise Autoregressive Policy Learning via Flow Matching [11.405254375363818]
  Hierarchical Flow Policy (HiFlow) is a tokenization-free coarse-to-fine autoregressive policy.<n>HiFlow constructs multi-scale continuous action targets from each action chunk via simple temporal pooling.<n>Experiments on MimicGen, RoboTwin 2.0, and real-world environments demonstrate that HiFlow consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2026-03-28T14:22:58Z)
• Latent Policy Steering through One-Step Flow Policies [34.06099184809882]
  offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration.<n>Latent Policy Steering (LPS) enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy.<n>Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.
  arXiv  Detail & Related papers  (2026-03-05T15:38:08Z)
• Flow Policy Gradients for Robot Control [67.61978635211048]
  Flow matching policy gradients can be made effective for training and fine-tuning more expressive policies.<n>We show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.
  arXiv  Detail & Related papers  (2026-02-02T18:56:49Z)
• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• Dense Policy: Bidirectional Autoregressive Learning of Actions [51.60428100831717]
  This paper introduces a bidirectionally expanded learning approach, termed Dense Policy, to establish a new paradigm for autoregressive policies in action prediction.<n>It employs a lightweight encoder-only architecture to iteratively unfold the action sequence from an initial single frame into the target sequence in a coarse-to-fine manner.<n>Experiments validate that our dense policy has superior autoregressive learning capabilities and can surpass existing holistic generative policies.
  arXiv  Detail & Related papers  (2025-03-17T14:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.