Deep Reinforcement Learning for Robotic Manipulation under Distribution Shift with Bounded Extremum Seeking

• URL: http://arxiv.org/abs/2604.01142v1
• Date: Wed, 01 Apr 2026 16:59:01 GMT
• Title: Deep Reinforcement Learning for Robotic Manipulation under Distribution Shift with Bounded Extremum Seeking
• Abstract summary: Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution.<n>This limitation is especially important in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time.<n>In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking to improve robustness under such conditions.
• Score: 42.278434352794676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning has shown strong performance in robotic manipulation, but learned policies often degrade in performance when test conditions differ from the training distribution. This limitation is especially important in contact-rich tasks such as pushing and pick-and-place, where changes in goals, contact conditions, or robot dynamics can drive the system out-of-distribution at inference time. In this paper, we investigate a hybrid controller that combines reinforcement learning with bounded extremum seeking to improve robustness under such conditions. In the proposed approach, deep deterministic policy gradient (DDPG) policies are trained under standard conditions on the robotic pushing and pick-and-place tasks, and are then combined with bounded ES during deployment. The RL policy provides fast manipulation behavior, while bounded ES ensures robustness of the overall controller to time variations when operating conditions depart from those seen during training. The resulting controller is evaluated under several out-of-distribution settings, including time-varying goals and spatially varying friction patches.

Related papers

• Safety-aware Skill Adaptation for Reinforcement Learning in Dynamic Environments [2.9923891863939946]
  Dist-GPRL is a distance-aware and safety-guided reinforcement learning framework for structured robot skill adaptation.<n>Building upon Gaussian Process (GP)-based skill parameterisation, our framework sequentially adapts overlapping local windows of sparse trajectory via-points.<n>We evaluate the framework on two dynamic object-manipulation tasks in simulation and transfer the learned policy to real-world robot execution.
  arXiv  Detail & Related papers  (2026-09-10T12:05:54Z)
• Steering Generative Reinforcement Learning into Stable Robotic Controller [40.70263465163786]
  SteerGenPO is a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller.<n>We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task.
  arXiv  Detail & Related papers  (2026-06-15T11:16:37Z)
• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers [5.690649768462432]
  We present ContractionPPO, a framework for robust planning and control of legged robots.<n>ContractionPPO produces a contraction metric that certifies exponential stability of the simulated closed-loop system.<n>Our hardware experiments on quadruped locomotion demonstrate that ContractionPPO enables robust, certifiably stable control even under strong external perturbations.
  arXiv  Detail & Related papers  (2026-03-20T04:32:18Z)
• GPO: Growing Policy Optimization for Legged Robot Locomotion and Whole-Body Control [20.055658070161183]
  Training reinforcement learning (RL) policies for legged robots remains challenging due to high-dimensional continuous actions, hardware constraints, and limited exploration.<n>We introduce Growing Policy Optimization (GPO), a training framework that applies a time-varying action transformation to restrict the effective action space in the early stage.<n>We evaluate GPO on both quadruped and hexapod robots, including zero-shot deployment of simulation-trained policies on hardware.
  arXiv  Detail & Related papers  (2026-01-28T14:49:52Z)
• Learning Deterministic Policies with Policy Gradients in Constrained Markov Decision Processes [59.27926064817273]
  We introduce an exploration-agnostic algorithm, called C-PG, which enjoys global last-iterate convergence guarantees under domination assumptions.<n>We empirically validate both the action-based (C-PGAE) and parameter-based (C-PGPE) variants of C-PG on constrained control tasks.
  arXiv  Detail & Related papers  (2025-06-06T10:29:05Z)
• SATA: Safe and Adaptive Torque-Based Locomotion Policies Inspired by Animal Learning [10.138425472807368]
  SATA is a bio-inspired framework that mimics key biomechanical principles and adaptive learning mechanisms observed in animal locomotion.<n>Our approach effectively addresses the inherent challenges of learning torque-based policies by significantly improving early-stage exploration.<n>Our experimental results indicate that SATA demonstrates remarkable compliance and safety, even in challenging environments.
  arXiv  Detail & Related papers  (2025-02-18T09:25:37Z)
• COMBO-Grasp: Learning Constraint-Based Manipulation for Bimanual Occluded Grasping [56.907940167333656]
  Occluded robot grasping is where the desired grasp poses are kinematically infeasible due to environmental constraints such as surface collisions.<n>Traditional robot manipulation approaches struggle with the complexity of non-prehensile or bimanual strategies commonly used by humans.<n>We introduce Constraint-based Manipulation for Bimanual Occluded Grasping (COMBO-Grasp), a learning-based approach which leverages two coordinated policies.
  arXiv  Detail & Related papers  (2025-02-12T01:31:01Z)
• Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Tasks [0.24578723416255746]
  In robotics, contemporary strategies are learning-based, characterized by a complex black-box nature and a lack of interpretability. We propose integrating a collision-free trajectory planner based on deep reinforcement learning (DRL) with a novel auto-tuning low-level control strategy.
  arXiv  Detail & Related papers  (2024-02-04T15:54:03Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.