Revisiting Action Factorization for Complex Action Spaces

• URL: http://arxiv.org/abs/2606.26574v1
• Date: Thu, 25 Jun 2026 03:48:56 GMT
• Title: Revisiting Action Factorization for Complex Action Spaces
• Abstract summary: This article provides a cross-sectional study of factorization methods on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]<n>We find that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albiet both at increased computational cost.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many real-world control problems involve hybrid discrete-continuous action spaces. For example, steering and signaling in autonomous driving, and aiming and firing in robotics or video-games. Despite real-world hybrid factorization and reinforcement learning framework support for complex action spaces (e.g., Gymnasium, PettingZoo, TorchRL, SeedRL, Mujoco, etc), the default environments within those frameworks often implement uniform action space configurations (LunarLander, Walker2D, Cheetah, SMAC, SUMO, Ant, Atari). Landmark hybrid-action benchmarks (RoboCup 2D HFO, SC2LE, Platform, CARLA, etc) are mostly heavyweight or archival implementations originating from papers which test one or a small number of competing factorization methods on one kind of control. This article provides a cross-sectional study of factorization methods [independent networks, shared encoder, VDN, QPLEX, Joint, Auto-Regressive] on each of three families of algorithms [PPO, SAC, DQN] across three action spaces [discretized, hybrid, continuous] over four lightweight environments [Platform, hybrid-LunarLander, Hybrid-Shoot, CoopPush]. Accounting for some invalid pairings such as joint-continuous, we are left with 220 configurations to analyze each method. We provide two new C++ parallel gymnasium and petting-zoo compliant environments [CoopPush, Hybrid-Shoot] to isolate particular challenges such as state-dependent inter-action dependence. Finally, we introduce VDN-PPO and PPO-MIX which use a branching critic to assign credit to multi-headed PPO. These variants out-perform all other tested PPO factorizations. Our results suggest that branching dueling architectures balance compute and performance most effectively, with Auto-Regressive actions reaching the highest performance overall and native continuous SAC outperforming discrete and hybrid algorithms, albiet both at increased computational cost.

Related papers

• Rethinking Air-Ground Collaboration: A Progressive Cross-Task Benchmark and Socialized Learning Framework [12.881497228808078]
  Socialized Co-Perception ( SCP) organizes collaboration progressively from aerial global localization to ground target association and identity-aware parsing.<n> SCP achieves a 3.73% coevolutionary gain and a 7.86% improvement in average downstream performance.
  arXiv  Detail & Related papers  (2026-06-17T09:17:55Z)
• Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space [2.4382430407654767]
  We propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that handles parameterized hybrid action spaces in a principled manner.<n>We conduct a theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures.<n>We introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization.
  arXiv  Detail & Related papers  (2026-03-01T22:16:27Z)
• Multi-Agent DRL for V2X Resource Allocation: Disentangling Challenges and Benchmarking Solutions [51.22818149833102]
  Multi-agent reinforcement learning (MARL) has emerged as a promising approach for radio resource allocation in vehicle-to-everything (C-V2X) networks.<n>However, the multifaceted challenges inherent to MARL are often intertwined, making it difficult to understand their individual impact on performance in vehicular environments.<n>We bridge this gap by formulating C-V2X RRA as a sequence of multi-agent interference games with progressively increasing complexity.
  arXiv  Detail & Related papers  (2026-02-18T14:46:56Z)
• CHDP: Cooperative Hybrid Diffusion Policies for Reinforcement Learning in Parameterized Action Space [9.192754462575218]
  We propose a textbfCooperative Hybrid Diffusion Policies (CHDP) framework to solve the hybrid action space problem.<n>CHDP employs two cooperative agents that leverage a discrete and a continuous diffusion policy, respectively.<n>On challenging hybrid action benchmarks, CHDP outperforms the state-of-the-art method by up to $19.3%$ in success rate.
  arXiv  Detail & Related papers  (2026-01-09T09:50:47Z)
• UniMPR: A Unified Framework for Multimodal Place Recognition with Heterogeneous Sensor Configurations [14.975915291012983]
  We propose UniMPR, a unified framework for multimodal place recognition.<n>Using only one trained model, it can seamlessly adapt to any combination of common perceptual modalities.<n> Experiments on seven datasets demonstrate that UniMPR achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-20T09:01:18Z)
• QoS-Aware Hierarchical Reinforcement Learning for Joint Link Selection and Trajectory Optimization in SAGIN-Supported UAV Mobility Management [52.15690855486153]
  A space-air-ground integrated network (SAGIN) has emerged as an essential architecture for enabling ubiquitous UAV connectivity.<n>This paper formulates UAV mobility management in SAGIN as a constrained multiobjective joint optimization problem.
  arXiv  Detail & Related papers  (2025-12-17T06:22:46Z)
• Inter2Former: Dynamic Hybrid Attention for Efficient High-Precision Interactive [58.0729162588429]
  Interactive segmentation improves annotation efficiency by segmenting target regions from user prompts.<n>Current approaches face a critical trade-off: dense-token methods achieve superior accuracy but suffer from prohibitively slow processing on CPU devices.<n>We propose Inter2Former to address this challenge by optimizing computation allocation in dense-token processing.
  arXiv  Detail & Related papers  (2025-07-13T12:33:37Z)
• Reinforcement learning with combinatorial actions for coupled restless bandits [62.89013331120493]
  We propose SEQUOIA, an RL algorithm that directly optimize for long-term reward over the feasible action space.<n>We empirically validate SEQUOIA on four novel restless bandit problems with constraints: multiple interventions, path constraints, bipartite matching, and capacity constraints.
  arXiv  Detail & Related papers  (2025-03-01T21:25:21Z)
• Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies [3.224364956163198]
  Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially when deploying to hardware in the real-world. We identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline.
  arXiv  Detail & Related papers  (2024-10-22T02:21:30Z)
• Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration [87.53543137162488]
  We propose an easy-to-implement online reinforcement learning (online RL) framework called textttMEX. textttMEX integrates estimation and planning components while balancing exploration exploitation automatically. It can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards.
  arXiv  Detail & Related papers  (2023-05-29T17:25:26Z)
• HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation [30.621472051415857]
  Most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space. We propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. We evaluate HyAR in a variety of environments with discrete-continuous action space.
  arXiv  Detail & Related papers  (2021-09-12T11:26:27Z)
• Composable Learning with Sparse Kernel Representations [110.19179439773578]
  We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space. We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function. We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment.
  arXiv  Detail & Related papers  (2021-03-26T13:58:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.