Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies

• URL: http://arxiv.org/abs/2410.16632v1
• Date: Tue, 22 Oct 2024 02:21:30 GMT
• Title: Benchmarking Smoothness and Reducing High-Frequency Oscillations in Continuous Control Policies
• Abstract summary: Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially when deploying to hardware in the real-world. We identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline.
• Score: 3.224364956163198
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reinforcement learning (RL) policies are prone to high-frequency oscillations, especially undesirable when deploying to hardware in the real-world. In this paper, we identify, categorize, and compare methods from the literature that aim to mitigate high-frequency oscillations in deep RL. We define two broad classes: loss regularization and architectural methods. At their core, these methods incentivize learning a smooth mapping, such that nearby states in the input space produce nearby actions in the output space. We present benchmarks in terms of policy performance and control smoothness on traditional RL environments from the Gymnasium and a complex manipulation task, as well as three robotics locomotion tasks that include deployment and evaluation with real-world hardware. Finally, we also propose hybrid methods that combine elements from both loss regularization and architectural methods. We find that the best-performing hybrid outperforms other methods, and improves control smoothness by 26.8% over the baseline, with a worst-case performance degradation of just 2.8%.

Related papers

• CLEAR: Conflict-aware Learning via Evidence-guided Adaptive Routing for Unified Sparse-View 3D Gaussian Super-Resolution [80.43237927269575]
  We propose CLEAR as the first unified single-stage framework for Sparse-view 3D Gaussian Splatting Super-resolution.<n> CLEAR performs joint the optimization of authentic LR observations and external HR priors within a unified Gaussian representation.<n>Experiments on both synthetic and real-world $4times$ super-resolution benchmarks demonstrate that CLEAR consistently achieves state-of-the-art rendering quality.
  arXiv  Detail & Related papers  (2026-08-03T13:31:53Z)
• Neural Backward Reach-Avoid Tubes with MPC Supervision for High-Dimensional Systems: An Application to Safe Spacecraft Docking [6.387263468033964]
  Hamilton-Jacobi (HJ) reachability provides formal reach-avoid guarantees, but classical solvers are limited to low-dimensional systems.<n>We propose a learning-based Backward Reach-Avoid Tube (BRAT) framework that tightly integrates HJ structure with MPC-based supervision.<n>We evaluate the proposed method on a 6D planar docking problem against grid-based ground truth and then scale to the full 13D system.
  arXiv  Detail & Related papers  (2026-05-03T19:00:06Z)
• Hybrid TD3: Overestimation Bias Analysis and Stable Policy Optimization for Hybrid Action Space [2.4382430407654767]
  We propose Hybrid TD3, an extension of Twin Delayed Deep Deterministic Policy Gradient (TD3) that handles parameterized hybrid action spaces in a principled manner.<n>We conduct a theoretical analysis of overestimation bias in hybrid action settings, deriving formal bounds under twin-critic architectures.<n>We introduce a weighted clipped Q-learning target that marginalizes over the discrete action distribution, achieving equivalent bias reduction to standard clipped minimization.
  arXiv  Detail & Related papers  (2026-03-01T22:16:27Z)
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training [38.8378349968766]
  Reinforcement Learning with Verifiable Rewards (RLVR) is highly dependent on high-quality labeled data.<n>Existing unsupervised RLVR methods, including pure entropy minimization, can overfit to incorrect labels.<n>We propose a novel two-stage, token-level entropy optimization method for RLVR.
  arXiv  Detail & Related papers  (2025-11-11T01:42:37Z)
• Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learning [22.966488300685484]
  Reinforcement learning (RL) has made significant advancements, achieving performance in various tasks.<n>Many environments are inherently non-stationary. This non-stationarity results in the requirement of millions of iterations, leading to low sample efficiency.<n>We introduce the Clustering Orthogonal Modified Weight (COWM) layer, which can be integrated into the policy network of any RL algorithm and mitigate non-stationarity effectively.
  arXiv  Detail & Related papers  (2025-11-02T13:45:01Z)
• Towards Accurate and Efficient Waste Image Classification: A Hybrid Deep Learning and Machine Learning Approach [0.05599792629509228]
  This study provides a comprehensive comparison of three paradigms: (1) machine learning algorithms using handcrafted features, (2) deep learning architectures, including ResNet variants and EfficientNetV2S, and (3) a hybrid approach that utilizes deep models for feature extraction combined with classical classifiers such as Support Vector Machine and Logistic Regression to identify the most effective strategy.<n>Experiments on three public datasets - TrashNet, Garbage Classification, and a refined Household Garbage dataset- demonstrate that the hybrid method consistently outperforms the others, achieving up to 100% accuracy on TrashNet and the refined Household set, and 99.87% on Garbage Classification, thereby surpassing state-of-the
  arXiv  Detail & Related papers  (2025-10-22T08:32:51Z)
• HYPERDOA: Robust and Efficient DoA Estimation using Hyperdimensional Computing [8.27483835715597]
  We introduce HYPERDOA, a novel estimator leveraging Hyperdimensional Computing (HDC)<n>It achieves 35.39% higher accuracy than state-of-the-art methods in low-SNR, coherent-source scenarios.<n>It also consumes 93% less energy than competing neural baselines on an embedded NVIDIA Jetson Xavier NX platform.
  arXiv  Detail & Related papers  (2025-10-12T17:42:01Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• Decoupled Hierarchical Reinforcement Learning with State Abstraction for Discrete Grids [3.772834044395258]
  This paper presents a decoupled hierarchical RL framework integrating state abstraction (DcHRL-SA)<n> Experiments conducted in two customized grid environments demonstrate that the proposed approach consistently outperforms PPO in terms of exploration efficiency, convergence speed, cumulative reward, and policy stability.
  arXiv  Detail & Related papers  (2025-06-01T06:36:19Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions. Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans. We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)
• CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishing [5.467140383171385]
  This work presents an experimental demonstration of the hybrid RL algorithm CHEQ for robotic polishing with variable impedance. On hardware, CHEQ achieves effective polishing behaviour, requiring only eight hours of training and incurring just five failures. Results highlight the potential of adaptive hybrid RL for real-world, contact-rich tasks trained directly on hardware.
  arXiv  Detail & Related papers  (2025-01-14T10:13:41Z)
• Improving generalization of robot locomotion policies via Sharpness-Aware Reinforcement Learning [0.5399800035598186]
  Differentiable simulators offer improved sample efficiency through exact gradients, but can be unstable in contact-rich environments. This paper introduces a novel approach integrating sharpness-aware optimization into gradient-based reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2024-11-29T14:25:54Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Adjustable Robust Reinforcement Learning for Online 3D Bin Packing [11.157035538606968]
  Current deep reinforcement learning methods for online 3D-BPP fail in real-world settings where some worst-case scenarios can materialize. We propose an adjustable robust reinforcement learning framework that allows efficient adjustment of robustness weights. Experiments demonstrate that AR2L is versatile in the sense that it improves policy robustness while maintaining an acceptable level of performance for the nominal case.
  arXiv  Detail & Related papers  (2023-10-06T15:34:21Z)
• DTC: Deep Tracking Control [16.2850135844455]
  We propose a hybrid control architecture that combines the advantages of both worlds to achieve greater robustness, foot-placement accuracy, and terrain generalization. A deep neural network policy is trained in simulation, aiming to track the optimized footholds. We demonstrate superior robustness in the presence of slippery or deformable ground when compared to model-based counterparts.
  arXiv  Detail & Related papers  (2023-09-27T07:57:37Z)
• A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale [64.10124092250126]
  Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.
  arXiv  Detail & Related papers  (2023-04-19T18:09:27Z)
• Re-Evaluating LiDAR Scene Flow for Autonomous Driving [80.37947791534985]
  Popular benchmarks for self-supervised LiDAR scene flow have unrealistic rates of dynamic motion, unrealistic correspondences, and unrealistic sampling patterns. We evaluate a suite of top methods on a suite of real-world datasets. We show that despite the emphasis placed on learning, most performance gains are caused by pre- and post-processing steps.
  arXiv  Detail & Related papers  (2023-04-04T22:45:50Z)
• Accelerated Policy Learning with Parallel Differentiable Simulation [59.665651562534755]
  We present a differentiable simulator and a new policy learning algorithm (SHAC) Our algorithm alleviates problems with local minima through a smooth critic function. We show substantial improvements in sample efficiency and wall-clock time over state-of-the-art RL and differentiable simulation-based algorithms.
  arXiv  Detail & Related papers  (2022-04-14T17:46:26Z)
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning [102.78958681141577]
  We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
  arXiv  Detail & Related papers  (2020-07-09T17:08:44Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy [3.404507240556492]
  Evolution strategy (ES) has been shown great promise in many challenging reinforcement learning (RL) tasks. There are two limitations in the current ES practice that may hinder its otherwise further capabilities. In this work, we employ a Directional Gaussian Smoothing Evolutionary Strategy (DGS-ES) to accelerate RL training. We show that DGS-ES is highly scalable, possesses superior wall-clock time, and achieves competitive reward scores to other popular policy gradient and ES approaches.
  arXiv  Detail & Related papers  (2020-02-21T01:05:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.