Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient

• URL: http://arxiv.org/abs/2605.26478v1
• Date: Tue, 26 May 2026 02:35:08 GMT
• Title: Efficient On-policy Visual-RL via Stochastic Decoupled Policy Gradient
• Abstract summary: We present the decoupled policy gradient (SDPG), a lightweight visual reinforcement learning (RL) method that trains diverse visuomotor control policies end-to-end.<n>SDPG estimates policy gradients via random perturbations of trajectory rollouts, requiring orders of magnitude fewer batch-rendered environments.<n>On visual MuJoCo benchmarks, SDPG consistently outperforms baseline methods in training time, memory usage, and rewards.
• Score: 9.971505157889503
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present the stochastic decoupled policy gradient (SDPG), a lightweight visual reinforcement learning (RL) method that trains diverse visuomotor control policies end-to-end within a few hours on a single NVIDIA RTX 4080 GPU. SDPG estimates policy gradients via random perturbations of trajectory rollouts, requiring orders of magnitude fewer batch-rendered environments and substantially reducing compute and memory overhead. On visual MuJoCo benchmarks, SDPG consistently outperforms baseline methods in training time, memory usage, and rewards. Finally, to support future research, we introduce a suite of realistic visual robotics benchmarks spanning dexterous manipulation, challenging locomotion, and demonstrate effective sim-to-real transfer on physical hardware.

Related papers

• CLEAR: Closed-Loop Reinforcement Learning at Scale for End-to-End Autonomous Driving [66.77752555295355]
  We present CLEAR, a system that enables closed-loop training using Reinforcement Learning (RL) at scale for E2E-AD.<n>We show that with a simple reward, CLEAR significantly outperforms previous methods and sets new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-07-03T00:34:50Z)
• Towards End to End Motion Planning and Execution for Autonomous Underwater Vehicles Using Reinforcement Learning [2.1942030377331245]
  This paper explores the feasibility of an end-to-end Deep Reinforcement Learning (DRL) approach that maps raw sensor data directly to thruster commands.<n>A High-Level (HL) policy operating at 2Hz processes raw $84 times 84$ pixel monocular camera frames, stacked $100 times 100$ pixel forward-looking imaging sonar, and proprioceptive data to generate spatial subgoals.<n>A Low-Level (LL) policy operating at 10Hz converts these subgoals into thruster commands.
  arXiv  Detail & Related papers  (2026-06-07T08:34:29Z)
• Simple but Stable, Fast and Safe: Achieve End-to-end Control by High-Fidelity Differentiable Simulation [14.322775509774246]
  Obstacle avoidance is a fundamental vision-based task essential for enabling quadrotors to perform advanced applications.<n>We propose a novel end-to-end policy that directly maps depth images to low-level bodyrate commands by reinforcement learning.<n>Results demonstrate that the proposed approach achieves the highest success rate and the lowest jerk among state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-04-12T09:38:00Z)
• FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control [55.38832429564216]
  Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable.<n>We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic.<n>Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency.
  arXiv  Detail & Related papers  (2026-04-06T09:03:41Z)
• Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer [59.02729900344616]
  GPU-accelerated, photorealistic simulation has opened a scalable data-generation path for robot learning.<n>We develop a teacher-student-bootstrap learning framework for vision-based humanoid loco-manipulation.<n>This represents the first humanoid sim-to-real policy capable of diverse articulated loco-manipulation using pure RGB perception.
  arXiv  Detail & Related papers  (2025-11-30T20:07:13Z)
• TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning [63.73629127832652]
  We introduce TD-JEPA, which leverages TD-based latent-predictive representations into unsupervised RL.<n> TD-JEPA trains explicit state and task encoders, a policy-conditioned multi-step predictor, and a set of parameterized policies directly in latent space.<n> Empirically, TD-JEPA matches or outperforms state-of-the-art baselines on locomotion, navigation, and manipulation tasks across 13 datasets.
  arXiv  Detail & Related papers  (2025-10-01T10:21:18Z)
• Relative Entropy Pathwise Policy Optimization [66.03329137921949]
  We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories.<n>We show how to combine policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning.
  arXiv  Detail & Related papers  (2025-07-15T06:24:07Z)
• Accelerating Visual-Policy Learning through Parallel Differentiable Simulation [3.70729078195191]
  We propose a computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gradients.<n>Our approach decouples the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems.<n> Notably, our method achieves a $4times$ improvement in final return, and successfully learns a humanoid running policy within 4 hours on a single GPU.
  arXiv  Detail & Related papers  (2025-05-15T18:38:36Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [57.278726604424556]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers.<n>Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy.<n>We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• GPU-Accelerated Policy Optimization via Batch Automatic Differentiation of Gaussian Processes for Real-World Control [8.720903734757627]
  We develop a policy optimization method by leveraging fast predictive sampling methods to process batches of trajectories in every forward pass. We demonstrate the effectiveness of our approach in training policies on a set of reference-tracking control experiments with a heavy-duty machine.
  arXiv  Detail & Related papers  (2022-02-28T09:31:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.