Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

• URL: http://arxiv.org/abs/2605.14297v1
• Date: Thu, 14 May 2026 02:59:45 GMT
• Title: Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients
• Abstract summary: We study reinforcement learning in hybrid discrete-continuous action spaces.<n>We propose Hybrid Policy Optimization (HPO), which backpropagates through the simulator wherever smoothness permits.<n>HPO substantially outperforms PPO on inventory control and switched linear-quadratic regulator problems.
• Score: 3.1698277542519997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study reinforcement learning in hybrid discrete-continuous action spaces, such as settings where the discrete component selects a regime (or index) and the continuous component optimizes within it -- a structure common in robotics, control, and operations problems. Standard model-free policy gradient methods rely on score-function (SF) estimators and suffer from severe credit-assignment issues in high-dimensional settings, leading to poor gradient quality. On the other hand, differentiable simulation largely sidesteps these issues by backpropagating through a simulator, but the presence of discrete actions or non-smooth dynamics yields biased or uninformative gradients. To address this, we propose Hybrid Policy Optimization (HPO), which backpropagates through the simulator wherever smoothness permits, using a mixed gradient estimator that combines pathwise and SF gradients while maintaining unbiasedness. We also show how problems with action discontinuities can be reformulated in hybrid form, further broadening its applicability. Empirically, HPO substantially outperforms PPO on inventory control and switched linear-quadratic regulator problems, with performance gaps increasing as the continuous action dimension grows. Finally, we characterize the structure of the mixed gradient, showing that its cross term -- which captures how continuous actions influence future discrete decisions -- becomes negligible near a discrete best response, thereby enabling approximate decentralized updates of the continuous and discrete components and reducing variance near optimality. All resources are available at github.com/MatiasAlvo/hybrid-rl.

Related papers

• ReFPO: Reflow Regularization for Flow Matching Policy Gradients [58.32178725687043]
  We present a simple online RL method that adds explicit Reflow regularization to Flow Matching Policy Gradients.<n>We experimentally demonstrate that ReFPO improves average performance and discretization across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks.
  arXiv  Detail & Related papers  (2026-06-19T04:23:10Z)
• On-Average Stability of Multipass Preconditioned SGD and Effective Dimension [15.521925194920891]
  We study trade-offs between the population risk curvature, geometry of the noise, and preconditioning on the generalisation ability of the multipass Preconditioned Gradient Descent (PSGD)<n>In particular, we show that an improperly chosen preconditioner can yield suboptimal effective dimension dependence in both optimisation and generalisation.
  arXiv  Detail & Related papers  (2026-03-12T14:41:44Z)
• Score-Guided Proximal Projection: A Unified Geometric Framework for Rectified Flow Editing [1.0312968200748118]
  Rectified Flow models achieve state-of-the-art generation quality, yet controlling them for precise tasks remains a challenge.<n>Current approaches bifurcate into inversion-based guidance, which suffers from "geometric locking"<n>We propose Score-Guided Proximal Projection, a unified framework that bridges the gap between deterministic optimization and contraction sampling.
  arXiv  Detail & Related papers  (2026-03-05T23:44:45Z)
• Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space [2.4382430407654767]
  We propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that handles parameterized hybrid action spaces in a principled manner.<n>We conduct a theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures.<n>We introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization.
  arXiv  Detail & Related papers  (2026-03-01T22:16:27Z)
• Certified Gradient-Based Contact-Rich Manipulation via Smoothing-Error Reachable Tubes [5.119652071085673]
  We develop a novel gradient-based policy synthesis method for contact-rich manipulation.<n>We evaluate our method on several contact-rich tasks, including planar pushing, object rotation, and in-hand dexterous manipulation.<n>By bridging differentiable physics with set-valued robust control, our method is the first certifiable gradient-based policy synthesis method for contact-rich manipulation.
  arXiv  Detail & Related papers  (2026-02-10T03:19:42Z)
• Adaptive Linear Path Model-Based Diffusion [52.84663832658799]
  We introduce Linear Path Model-Based Diffusion (LP-MBD), which replaces the variance-preserving schedule with a flow-matching linear probability path.<n>We also propose Adaptive LP-MBD (ALP-MBD), which leverages reinforcement learning to adjust diffusion steps and noise levels according to task complexity and environmental conditions.
  arXiv  Detail & Related papers  (2026-02-02T21:33:03Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization [88.7827307535107]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face significant image quality degradation under a low-dimensional budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as EPD-EPr), a novel ODE solver that mitigates these errors by incorporating multiple gradient parallel evaluations in each step.
  arXiv  Detail & Related papers  (2025-12-28T05:48:55Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Optimization on manifolds: A symplectic approach [127.54402681305629]
  We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
  arXiv  Detail & Related papers  (2021-07-23T13:43:34Z)
• Improper Learning with Gradient-based Policy Optimization [62.50997487685586]
  We consider an improper reinforcement learning setting where the learner is given M base controllers for an unknown Markov Decision Process. We propose a gradient-based approach that operates over a class of improper mixtures of the controllers.
  arXiv  Detail & Related papers  (2021-02-16T14:53:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.