Learning Terrain-Aware Whole-Body Control for Perceptive Legged Loco-Manipulation

• URL: http://arxiv.org/abs/2605.31343v1
• Date: Fri, 29 May 2026 14:22:10 GMT
• Title: Learning Terrain-Aware Whole-Body Control for Perceptive Legged Loco-Manipulation
• Abstract summary: We introduce TA-WBC, a terrain-aware whole-body control framework for legged manipulators.<n>We employ a hybrid exteroception encoder to extract terrain features, providing an essential basis for the robot to proactively adapt posture and footholds.<n>To facilitate stable cross-terrain loco-manipulation, we propose a novel end-effector sampling method based on the foot contact plane.
• Score: 2.915167691607446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Legged manipulators integrate exceptional terrain adaptability along with mobile manipulation capabilities, which make them highly promising for deployment in human-centric environments. By coordinating the control of both legs and arms, a whole-body controller can significantly expand the operational workspace of legged manipulators. However, many existing whole-body controllers primarily depend on proprioception and do not incorporate the critical exteroception required for effective terrain topology perception. This limitation can hinder their ability to adapt to varying environmental conditions and navigate complex terrains effectively. In this paper, we introduce TA-WBC, a terrain-aware whole-body control framework for legged manipulators, which features a novel RL-based unified policy tailored to whole-body loco-manipulation tasks in various terrains. Specifically, we employ a hybrid exteroception encoder to extract terrain features, providing an essential basis for the robot to proactively adapt posture and footholds. Furthermore, to facilitate stable cross-terrain loco-manipulation, we propose a novel end-effector sampling method based on the foot contact plane, decoupling manipulation target from base fluctuations. Moreover, a dual-policy distillation module is introduced to integrate expansive whole-body motion with terrain adaptability without catastrophic forgetting. The simulation and real-world experiments validate the robustness of our proposed controller, which leads to a larger reachable space, less tracking error, and reduced unexpected stumbles. This unified policy highlights the promising capabilities of legged manipulators in performing loco-manipulation tasks across complex terrains.

Related papers

• Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking [78.30338456206984]
  We introduce a novel framework for training general-purpose 3D foothold-tracking policies.<n>By dynamically providing footstep support through a goal sampler, this method enables the learned policy to be agnostic to specific terrains.<n>Our new target representation effectively mitigates challenges arising in the real world, such as noisy and inaccurate pose estimation and foot contact estimation.
  arXiv  Detail & Related papers  (2026-06-06T16:44:56Z)
• CEER: Compliant End-Effector and Root Control as a Unified Interface for Hierarchical Humanoid Loco-Manipulation [11.545606851036787]
  We propose a compliant end-effector-root (EE-root) control abstraction for modular humanoid loco-manipulation.<n>EE-root provides a practical abstraction for humanoid loco-manipulation, enabling modular and scalable integration of diverse skills.
  arXiv  Detail & Related papers  (2026-05-19T15:23:40Z)
• ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation [55.467742403416175]
  We introduce a physics-driven neural algorithm that translates large-scale motion capture to humanoid embodiments.<n>We learn a unified multimodal controller that supports both dense references and sparse task specifications.<n>Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception.
  arXiv  Detail & Related papers  (2026-03-03T18:59:29Z)
• Pro-HOI: Perceptive Root-guided Humanoid-Object Interaction [57.252599851031505]
  We introduce Perceptive Root-guided Humanoid-Object Interaction, Pro-HOI.<n>Pro-HOI is a generalizable framework for robust humanoid loco-manipulation.<n> Empirical validation on a Unitree G1 robot demonstrates that Pro-HOI significantly outperforms baselines in generalization and robustness.
  arXiv  Detail & Related papers  (2026-03-01T14:19:59Z)
• PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes [18.916978556216097]
  PILOT is a single-stage reinforcement learning framework tailored for perceptive loco-manipulation.<n>It synergizes perceptive locomotion and expansive whole-body control within a single policy.<n> PILOT demonstrates superior stability, command tracking precision, and terrain traversability compared to existing baselines.
  arXiv  Detail & Related papers  (2026-01-24T12:25:20Z)
• CHIP: Adaptive Compliance for Humanoid Control through Hindsight Perturbation [70.5382178207975]
  hIsight Perturbation (CHIP) is a plug-and-play module that enables controllable end-effector stiffness.<n>CHIP is easy to implement and requires neither data augmentation nor additional reward tuning.<n>We show that a generalist motion-tracking controller trained with CHIP can perform a diverse set of forceful manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-16T18:56:04Z)
• Gait-Adaptive Perceptive Humanoid Locomotion with Real-Time Under-Base Terrain Reconstruction [86.53248703859718]
  We introduce a perceptive locomotion framework that merges terrain sensing, gait regulation, and whole-body control into a single reinforcement learning policy.<n>Experiments conducted on a 31-DoF, 1.65 m humanoid robot demonstrate robust locomotion in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2025-12-08T11:42:41Z)
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration [58.4036440289082]
  Hand-object motion-capture (MoCap) offer large-scale, contact-rich demonstrations and hold promise for dexterous robotic scopes.<n>We introduce Dexplore, a unified single-loop optimization that performs repositories and tracking to learn robot control policies directly from MoCap at scale.
  arXiv  Detail & Related papers  (2025-09-11T17:59:07Z)
• Nonprehensile Planar Manipulation through Reinforcement Learning with Multimodal Categorical Exploration [8.343657309038285]
  Reinforcement Learning is a powerful framework for developing such robot controllers. We propose a multimodal exploration approach through categorical distributions, which enables us to train planar pushing RL policies. We show that the learned policies are robust to external disturbances and observation noise, and scale to tasks with multiple pushers.
  arXiv  Detail & Related papers  (2023-08-04T16:55:00Z)
• Deep Whole-Body Control: Learning a Unified Policy for Manipulation and Locomotion [25.35885216505385]
  An attached arm can significantly increase the applicability of legged robots to mobile manipulation tasks. Standard hierarchical control pipeline for such legged manipulators is to decouple the controller into that of manipulation and locomotion. We learn a unified policy for whole-body control of a legged manipulator using reinforcement learning.
  arXiv  Detail & Related papers  (2022-10-18T17:59:30Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.