Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space

• URL: http://arxiv.org/abs/2603.01302v1
• Date: Sun, 01 Mar 2026 22:16:27 GMT
• Title: Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space
• Abstract summary: We propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that handles parameterized hybrid action spaces in a principled manner.<n>We conduct a theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures.<n>We introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization.
• Score: 2.4382430407654767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning in discrete-continuous hybrid action spaces presents fundamental challenges for robotic manipulation, where high-level task decisions and low-level joint-space execution must be jointly optimized. Existing approaches either discretize continuous components or relax discrete choices into continuous approximations, which suffer from scalability limitations and training instability in high-dimensional action spaces and under domain randomization. In this paper, we propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that natively handles parameterized hybrid action spaces in a principled manner. We conduct a rigorous theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures and establishing a complete bias ordering across five algorithmic variants. Building on this analysis, we introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization while improving policy smoothness. Experimental results demonstrate that Hybrid TD3 achieves superior training stability and competitive performance against state-of-the-art hybrid action baselines

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• Target-Aware Linear Regression Under Distribution Shift [2.705806052277982]
  Distribution shift between training and deployment is a pervasive challenge for modern AI systems.<n>We study this problem with a stable conditional mean $E[Ymid X]$ across source and target.<n>We identify the hybrid-loss estimator, which jointly incorporates both target marginals, as a benchmark target-aware estimator.
  arXiv  Detail & Related papers  (2026-06-22T02:24:50Z)
• D$^3$-MoE:Dual Disentangled Diffusion Mixture-of-Experts for Style-Controllable End-to-End Autonomous Driving [22.70855312181222]
  D$3$-MoE disentangles trajectory modeling along two complementary axes.<n>On the behavioral axis, generation is decoupled from selection.<n>On the physical axis, decoupled longitudinal and lateral routers activate their respective experts.
  arXiv  Detail & Related papers  (2026-06-03T13:46:46Z)
• Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients [3.1698277542519997]
  We study reinforcement learning in hybrid discrete-continuous action spaces.<n>We propose Hybrid Policy Optimization (HPO), which backpropagates through the simulator wherever smoothness permits.<n>HPO substantially outperforms PPO on inventory control and switched linear-quadratic regulator problems.
  arXiv  Detail & Related papers  (2026-05-14T02:59:45Z)
• LaST-R1: Reinforcing Robotic Manipulation via Adaptive Physical Latent Reasoning [90.86828952599147]
  We present LaST-R1, a novel reinforcement learning framework designed to harness "latent reasoning-before-acting" policies.<n>LaST-R1 achieves a near-perfect 99.9% average success rate on the LIBERO benchmark.<n>In real-world deployments, LaST-R1 yields up to a 22.5% average improvement over SOTA supervised fine-tuning approach.
  arXiv  Detail & Related papers  (2026-04-30T17:59:52Z)
• Coordinated Manipulation of Hybrid Deformable-Rigid Objects in Constrained Environments [3.915966095774856]
  This work presents a quasi-static optimization-based manipulation planner that employs a strain-based Cosserat rod model.<n>It exploits the compliance of deformable links to maneuver through constraints while achieving task-space objectives for the object that are unreachable with rigid tools.<n>The proposed algorithm is validated in simulation on various hDLO systems and experimentally on a three-link hDLO manipulated in a constrained environment using a dual-arm robotic system.
  arXiv  Detail & Related papers  (2026-03-13T12:34:49Z)
• SVLL: Staged Vision-Language Learning for Physically Grounded Embodied Task Planning [21.113678610046453]
  We present Staged Vision-Language Learning (SVLL), a unified three-stage framework for robust, physically-grounded embodied planning.<n>In the first two stages, SVLL decouples spatial grounding from temporal reasoning, establishing robust visual dependency before introducing sequential action history.<n>In the final stage, we identify a key limitation of standard Direct Preference Optimization (DPO), its purely relative nature -- optimizing only the preference gap between winning and losing trajectories.
  arXiv  Detail & Related papers  (2026-03-12T05:35:29Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• Constrained Dynamic Gaussian Splatting [47.982650444869336]
  Constrained Dynamic Gaussian Splatting (CDGS) is a novel framework that formulates dynamic scene reconstruction as a budget-constrained optimization problem.<n>We show that CDGS delivers optimal rendering quality under varying capacity limits, achieving over 3x compression compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-02-03T13:53:29Z)
• Steerable Adversarial Scenario Generation through Test-Time Preference Alignment [58.37104890690234]
  Adversarial scenario generation is a cost-effective approach for safety assessment of autonomous driving systems.<n>We introduce a new framework named textbfSteerable textbfAdversarial scenario textbfGEnerator (SAGE)<n>SAGE enables fine-grained test-time control over the trade-off between adversariality and realism without any retraining.
  arXiv  Detail & Related papers  (2025-09-24T13:27:35Z)
• Momentum-constrained Hybrid Heuristic Trajectory Optimization Framework with Residual-enhanced DRL for Visually Impaired Scenarios [4.735413508037063]
  This paper proposes a momentum-constrained hybrid trajectory optimization framework (MHHTOF) tailored for assistive navigation in visually impaired scenarios.<n>It integrates trajectory sampling generation, optimization and evaluation with residual deep reinforcement learning (DRL)<n> Experimental results demonstrate that the proposed LSTM-BResPPO achieves significantly faster convergence, attaining stable policy performance in approximately half the training required by the PPO.
  arXiv  Detail & Related papers  (2025-09-19T04:33:39Z)
• Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shifts [80.32933059529135]
  Test-Time Adaptation (TTA) methods have emerged to adapt to target distributions during inference.<n>We propose Dual Uncertainty Optimization (DUO), the first TTA framework designed to jointly minimize both uncertainties for robust M3OD.<n>In parallel, we design a semantic-aware normal field constraint that preserves geometric coherence in regions with clear semantic cues.
  arXiv  Detail & Related papers  (2025-08-28T07:09:21Z)
• Discrete-Guided Diffusion for Scalable and Safe Multi-Robot Motion Planning [56.240199425429445]
  Multi-Robot Motion Planning (MPMP) involves generating trajectories for multiple robots operating in a shared continuous workspace.<n>While discrete multi-agent finding (MAPF) methods are broadly adopted due to their scalability, their coarse discretization trajectory quality.<n>This paper tackles limitations of two approaches by introducing discrete MAPF solvers with constrained generative diffusion models.
  arXiv  Detail & Related papers  (2025-08-27T17:59:36Z)
• Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies [3.224364956163198]
  Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially when deploying to hardware in the real-world. We identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline.
  arXiv  Detail & Related papers  (2024-10-22T02:21:30Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.