StructRL: Structured Action-Space Exploration for Flow-Based VLAs

• URL: http://arxiv.org/abs/2608.15139v1
• Date: Sat, 15 Aug 2026 09:31:29 GMT
• Title: StructRL: Structured Action-Space Exploration for Flow-Based VLAs
• Abstract summary: Flow-based Vision-Language-Action models are now widely used for continuous robotic manipulation.<n>Online reinforcement learning is emerging as a key technique for adapting them to new tasks.<n>We show that effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups.
• Score: 81.43458629514139
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flow-based Vision-Language-Action (VLA) models are now widely used for continuous robotic manipulation, and online reinforcement learning (RL) is emerging as a key technique for adapting them to new tasks. Existing RL methods typically inject stochasticity inside the denoising chain, often through isotropic or temporally independent noise. However, effective robot exploration calls for structured noise: temporally smooth and scaled differently across action groups. We show that simply switching the in-chain noise to a structured form does not suffice: noise added at an intermediate flow time can be weakened by the remaining denoising steps before execution, a phenomenon we call \emph{Structured Noise Dilution}. We propose \textbf{StructRL}, which avoids dilution by relocating policy stochasticity to the action space via three coupled choices: (i) a deterministic ODE decoder, (ii) structured noise injected directly in the action space, and (iii) last-step replay, where policy-gradient updates avoid assigning likelihoods to intermediate denoising states. This keeps structured exploration tied to the executed action while providing a tractable training signal for the flow decoder. Across three flow-based VLA models on multiple simulated manipulation benchmarks and two real-world tasks, StructRL improves exploration efficiency and OOD performance over prior in-chain baselines, demonstrating the effectiveness of structured action-space exploration for adapting flow-based VLA with RL. \textbf{Project page:} https://flyfaerss.github.io/structrl/

Related papers

• Beyond Noise Steering: Dual-Latent Space Reinforcement Learning for Generative Robot Policy [13.530890681678619]
  Pretrained generative robot policies learn expressive action prior to demonstrations.<n>Existing reinforcement learning methods only steer the noisy space but fail to modulate intermediate action representations during the generation process.<n>We propose a novel Dual-Latent Space Reinforcement Learning framework, which complements initial-noise steering with representation-level control inside the frozen generator.
  arXiv  Detail & Related papers  (2026-09-10T09:04:04Z)
• HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RL [56.014913786737104]
  We introduce HAF (Humanoid Adaptation Framework) to transfer off-the-shelf generalist VLA foundation models to humanoid whole-body loco-manipulation.<n> HAF-VLA is a hierarchical action-flow generator built on a pretrained flow-matching VLA.<n>HaF-Steer is a latent offline-to-online RL pipeline that leverages flow-matching invertibility and DCT-based dimensionality reduction.
  arXiv  Detail & Related papers  (2026-08-17T17:22:33Z)
• DRIFT: Derailing Denoising Trajectories of Flow-Matching VLAs with Adversarial Patch Attack [23.73320512301939]
  Flow-matching vision-language-action (VLA) models such as pi0 generate robot actions by integrating a learned denoising velocity field.<n>We show that this robustness is largely illusory: it stems from prior attacks ignoring the multi-step denoising ODE.<n>We introduce DRIFT, a test-time universal adversarial patch placed on the robot's gripper that attacks the denoising velocity field of an off-the-shelf policy.
  arXiv  Detail & Related papers  (2026-08-04T06:47:26Z)
• TBD-VLA: Temporal Block Diffusion Vision Language Action Model [7.861095039299131]
  We introduce TBD-VLA, a discrete token-based VLA framework that incorporates block diffusion to enable temporal action generation.<n>We partition action sequences into temporal blocks and perform masked discrete diffusion within each block, while maintaining autoregressive generation across blocks.<n>This design unifies temporal autoregression and parallel action decoding, achieving both strong temporal coherence and improved inference speed.
  arXiv  Detail & Related papers  (2026-06-05T23:10:43Z)
• Action-to-Action Flow Matching [25.301629044539325]
  Diffusion-based policies have recently achieved remarkable success in robotics by formulating action prediction as a conditional denoising process.<n>We propose Action-to-Action flow matching (A2A), a novel policy paradigm that shifts from random sampling to initialization informed by the previous action.<n>A2A enables high-quality action generation in as few as a single inference step (0.56 ms latency), and exhibits superior robustness to visual perturbations and enhanced generalization to unseen configurations.
  arXiv  Detail & Related papers  (2026-02-07T02:39:49Z)
• KAN We Flow? Advancing Robotic Manipulation with 3D Flow Matching via KAN & RWKV [15.840121055564405]
  KAN-We-Flow is a flow-matching policy that builds a lightweight backbone for 3D manipulation.<n>Our design reduces parameters by 86.8%, maintains fast runtime, and achieves state-of-the-art success rates on Adroit, Meta-World, and DexArt benchmarks.
  arXiv  Detail & Related papers  (2026-02-01T09:27:56Z)
• Noise-Robust Tiny Object Localization with Flows [63.60972031108944]
  We propose a noise-robust localization framework leveraging normalizing flows for flexible error modeling and uncertainty-guided optimization.<n>Our method captures complex, non-Gaussian prediction distributions through flow-based error modeling, enabling robust learning under noisy supervision.<n>An uncertainty-aware gradient modulation mechanism further suppresses learning from high-uncertainty, noise-prone samples, mitigating overfitting while stabilizing training.
  arXiv  Detail & Related papers  (2026-01-02T09:16:55Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• NinA: Normalizing Flows in Action. Training VLA Models with Normalizing Flows [75.70583906344815]
  Diffusion models have been widely adopted as action decoders due to their ability to model complex, multimodal action distributions.<n>We present NinA, a fast and expressive alternative to diffusion-based decoders for Vision-Language-Action (VLA) models.
  arXiv  Detail & Related papers  (2025-08-23T00:02:15Z)
• SuperFlow++: Enhanced Spatiotemporal Consistency for Cross-Modal Data Pretraining [62.433137130087445]
  SuperFlow++ is a novel framework that integrates pretraining and downstream tasks using consecutive camera pairs.<n>We show that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions.<n>With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-25T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.