Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?

• URL: http://arxiv.org/abs/2604.18161v1
• Date: Mon, 20 Apr 2026 12:23:48 GMT
• Title: Does "Do Differentiable Simulators Give Better Policy Gradients?'' Give Better Policy Gradients?
• Abstract summary: discontinuous dynamics cause bias and undermine the effectiveness of 1st-order estimators.<n>We introduce DDCG, a lightweight test that switches estimators in nonsmooth regions.<n>We also present IVW-H, a per-step inverse-variance implementation that stabilizes variance without explicit discontinuity detection.
• Score: 25.53040167917892
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In policy gradient reinforcement learning, access to a differentiable model enables 1st-order gradient estimation that accelerates learning compared to relying solely on derivative-free 0th-order estimators. However, discontinuous dynamics cause bias and undermine the effectiveness of 1st-order estimators. Prior work addressed this bias by constructing a confidence interval around the REINFORCE 0th-order gradient estimator and using these bounds to detect discontinuities. However, the REINFORCE estimator is notoriously noisy, and we find that this method requires task-specific hyperparameter tuning and has low sample efficiency. This paper asks whether such bias is the primary obstacle and what minimal fixes suffice. First, we re-examine standard discontinuous settings from prior work and introduce DDCG, a lightweight test that switches estimators in nonsmooth regions; with a single hyperparameter, DDCG achieves robust performance and remains reliable with small samples. Second, on differentiable robotics control tasks, we present IVW-H, a per-step inverse-variance implementation that stabilizes variance without explicit discontinuity detection and yields strong results. Together, these findings indicate that while estimator switching improves robustness in controlled studies, careful variance control often dominates in practical deployments.

Related papers

• Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning [7.604367783364084]
  Collaborative Weighting Actor-Critic (CWAC) is a unified framework that accounts for predictive uncertainty in value estimation.<n>CWAC can be seamlessly integrated into existing off-policy algorithm frameworks with minimal overhead.
  arXiv  Detail & Related papers  (2026-07-29T06:18:19Z)
• Robust Activation Map Rectification for Weakly Supervised Volumetric Segmentation: Temporal Coherence as a Free Lunch [31.156115062163412]
  Weakly supervised segmentation relies heavily on class activation maps (CAMs) to initially localize target regions.<n>Existing remedies typically introduce additional training stages or prototype learning, increasing computational cost and reducing robustness.<n>We propose a training-free prototype-free framework that rectifies unreliable CAMs by exploiting temporal and structural coherence in volumetric data as a free lunch.
  arXiv  Detail & Related papers  (2026-07-22T08:07:38Z)
• Variational Neural Belief Parameterizations for Robust Dexterous Grasping under Multimodal Uncertainty [5.4872967954342675]
  Risk-sensitive POMDPs use particle-filter beliefs that scale poorly, obstruct gradient-based optimization, and estimate Conditional Value-at-Risk.<n>We instead formulate grasp acquisition as variational inference over latent contact parameters and object pose, representing the belief with a differentiable Gaussian mixture.
  arXiv  Detail & Related papers  (2026-04-28T17:40:49Z)
• A Unified Noise-Curvature View of Loss of Trainability [8.602734307457387]
  Loss of trainability (LoT) in continual learning occurs when steps no longer yield improvement as tasks evolve.<n>We introduce two complementary criteria: a batch-size-aware gradient-noise bound and a curvature volatility-controlled bound.<n>Using this threshold, we build a simple per-layer scheduler that keeps each layers effective step below a safe limit.
  arXiv  Detail & Related papers  (2025-09-24T02:11:13Z)
• Stable Neighbor Denoising for Source-free Domain Adaptive Segmentation [91.83820250747935]
  Pseudo-label noise is mainly contained in unstable samples in which predictions of most pixels undergo significant variations during self-training. We introduce the Stable Neighbor Denoising (SND) approach, which effectively discovers highly correlated stable and unstable samples. SND consistently outperforms state-of-the-art methods in various SFUDA semantic segmentation settings.
  arXiv  Detail & Related papers  (2024-06-10T21:44:52Z)
• C-Learner: Constrained Learning for Causal Inference [4.370964009390564]
  We propose a novel debiasing approach that achieves the best weighting of both worlds, producing stable plug-in estimates.<n>Our constrained learning framework solves for the best plug-in estimator under the constraint that the first-order error with respect to the plugged-in quantity is zero.
  arXiv  Detail & Related papers  (2024-05-15T16:38:28Z)
• Estimating the Hessian Matrix of Ranking Objectives for Stochastic Learning to Rank with Gradient Boosted Trees [63.18324983384337]
  We introduce the first learning to rank method for Gradient Boosted Decision Trees (GBDTs) Our main contribution is a novel estimator for the second-order derivatives, i.e., the Hessian matrix. We incorporate our estimator into the existing PL-Rank framework, which was originally designed for first-order derivatives only.
  arXiv  Detail & Related papers  (2024-04-18T13:53:32Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Differentiable Annealed Importance Sampling and the Perils of Gradient Noise [68.44523807580438]
  Annealed importance sampling (AIS) and related algorithms are highly effective tools for marginal likelihood estimation. Differentiability is a desirable property as it would admit the possibility of optimizing marginal likelihood as an objective. We propose a differentiable algorithm by abandoning Metropolis-Hastings steps, which further unlocks mini-batch computation.
  arXiv  Detail & Related papers  (2021-07-21T17:10:14Z)
• Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator [93.05919133288161]
  We show that the variance of the straight-through variant of the popular Gumbel-Softmax estimator can be reduced through Rao-Blackwellization. This provably reduces the mean squared error. We empirically demonstrate that this leads to variance reduction, faster convergence, and generally improved performance in two unsupervised latent variable models.
  arXiv  Detail & Related papers  (2020-10-09T22:54:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.