Where to Touch, How to Contact: Hierarchical RL-MPC Framework for Geometry-Aware Long-Horizon Dexterous Manipulation

• URL: http://arxiv.org/abs/2601.10930v2
• Date: Fri, 23 Jan 2026 18:00:43 GMT
• Title: Where to Touch, How to Contact: Hierarchical RL-MPC Framework for Geometry-Aware Long-Horizon Dexterous Manipulation
• Abstract summary: A key challenge in contact-rich dexterous manipulation is the need to jointly reason over geometry, kinematic constraints, and intricate, nonsmooth contact dynamics.<n>We propose a hierarchical RL--MPC framework in which a high-level reinforcement learning (RL) policy predicts a contact intention.<n>We evaluate the framework on non-prehensile tasks, including geometry-generalized pushing and object 3D reorientation.
• Score: 15.309183490293721
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key challenge in contact-rich dexterous manipulation is the need to jointly reason over geometry, kinematic constraints, and intricate, nonsmooth contact dynamics. End-to-end visuomotor policies bypass this structure, but often require large amounts of data, transfer poorly from simulation to reality, and generalize weakly across tasks/embodiments. We address those limitations by leveraging a simple insight: dexterous manipulation is inherently hierarchical - at a high level, a robot decides where to touch (geometry) and move the object (kinematics); at a low level it determines how to realize that plan through contact dynamics. Building on this insight, we propose a hierarchical RL--MPC framework in which a high-level reinforcement learning (RL) policy predicts a contact intention, a novel object-centric interface that specifies (i) an object-surface contact location and (ii) a post-contact object-level subgoal pose. Conditioned on this contact intention, a low-level contact-implicit model predictive control (MPC) optimizes local contact modes and replans with contact dynamics to generate robot actions that robustly drive the object toward each subgoal. We evaluate the framework on non-prehensile tasks, including geometry-generalized pushing and object 3D reorientation. It achieves near-100% success with substantially reduced data (10x less than end-to-end baselines), highly robust performance, and zero-shot sim-to-real transfer.

Related papers

• WM-Craftnet: World Synesthesia Model for Generalizable and Robust Dexterous In-Hand Manipulation [7.577131730633025]
  Generalizable and robust in-hand manipulation requires a policy to infer object pose, geometry, contact, and potential slip.<n>We propose WM-Craftnet, a world-model-conditioned framework that learns compact action-conditioned latent dynamics.
  arXiv  Detail & Related papers  (2026-09-07T03:48:06Z)
• One Demonstration, Many Objects: Generalizing Manipulation via Local Contact Geometry [49.066054862620014]
  Learning from human demonstrations has emerged as a scalable alternative to robot teleoperation.<n>Recent sim-to-real RL methods incorporate such priors, but often omit rewards that explicitly incentivize precise contact.<n>We propose DemoMimic, a policy that manipulates objects by focusing on their geometry local to the contact points.
  arXiv  Detail & Related papers  (2026-09-01T23:04:35Z)
• GIRAF: Towards Generalizable Human Interactions with Articulated Objects [50.52795413484619]
  We present a text-conditioned diffusion model that addresses the challenge of Synthesizing realistic full-body human interactions with articulated objects.<n>Our method demonstrated strong generalization to unseen object configurations, surpassing current state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-07-08T19:30:44Z)
• ConCent: Contact-Centric Real-to-Sim-to-Real Learning from One Demonstration [17.067367964368486]
  Sim-to-real policy transfer is a promising paradigm for scaling robot manipulation without large-scale real-world data.<n>We propose a contact-centric real-to-sim-to-real RL framework that uses task-relevant contact event sequences extracted from real demonstrations.<n>Experiments on contact-rich manipulation tasks demonstrate more stable and robust sim-to-real policy transfer compared to unconstrained RL baselines.
  arXiv  Detail & Related papers  (2026-06-29T13:16:33Z)
• RelAfford6D: Relational 6D Affordance Graphs for Constraint-Driven Robotic Manipulation [15.618446008590409]
  We introduce RelAfford6, a training-free framework for complex articulated objects.<n>Our system deduces a semantic topology linking a primary interacting part to its physical anchor.<n>We analytically formulate downstream execution as a satisfaction problem.
  arXiv  Detail & Related papers  (2026-06-25T13:42:26Z)
• Beyond Binary: Sim-to-Real Dexterous Manipulation with Physics-Grounded Contact Representation [51.463992799813816]
  A primary bottleneck in contact-rich manipulation is the difficulty of collecting real-world data.<n>We introduce Center-of-Pressure, an effective tactile representation grounded in physical principles.<n>We evaluate CoP on two blind, challenging contact-rich manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-27T17:59:02Z)
• MaMi-HOI: Harmonizing Global Kinematics and Local Geometry for Human-Object Interaction Generation [14.056276184323321]
  MaMi-HOI reconciles textbfMacro-level fluidity with textbfMicro-level spatial precision.<n>Experiments validate that MaMi-HOI simultaneously achieves natural motion and precise contact.
  arXiv  Detail & Related papers  (2026-05-07T06:52:14Z)
• OmniVTA: Visuo-Tactile World Modeling for Contact-Rich Robotic Manipulation [60.609604885961716]
  We present textbfOmniViTac, a large-scale visuo-tactile-action dataset comprising $21,000+$ trajectories across $86$ tasks and $100+$ objects.<n>We propose textbf OmniVTA, a world-model-based visuo-tactile manipulation framework that integrates four tightly coupled modules.
  arXiv  Detail & Related papers  (2026-03-19T17:52:42Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• Semantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation [82.63833405368159]
  Generalizing tool manipulation requires both semantic planning and precise physical control.<n>We propose Semantic-Contact Fields (SCFields), a unified 3D representation fusing visual semantics with dense contact estimates.<n> Experiments on scraping, crayon drawing, and peeling demonstrate robust category-level generalization.
  arXiv  Detail & Related papers  (2026-02-14T16:05:08Z)
• HAIC: Humanoid Agile Object Interaction Control via Dynamics-Aware World Model [56.4392302336014]
  We present HAIC, a framework for robust interaction across diverse object dynamics without external state estimation.<n>Our key contribution is a dynamics predictor that estimates high-order object states (velocity, acceleration) solely from proprioceptive history.<n> Experiments on a humanoid robot show HAIC achieves high success rates in agile tasks.
  arXiv  Detail & Related papers  (2026-02-12T09:34:35Z)
• Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation [14.221542785249524]
  We introduce DreamTacVLA, a framework that grounds VLA models in contact physics by learning to feel the future.<n>Our model adopts a hierarchical perception scheme in which high-resolution tactile images serve as micro-vision inputs.<n>To further deepen the model's understanding of fine-grained contact dynamics, we finetune the system with a tactile world model that predicts future tactile signals.
  arXiv  Detail & Related papers  (2025-12-29T21:06:33Z)
• Decoupled Generative Modeling for Human-Object Interaction Synthesis [35.78156236836254]
  Existing approaches often require manually specified intermediate waypoints and place all optimization objectives on a single network.<n>We propose Decoupled Generative Modeling for Human-Object Interaction Synthesis (DecHOI)<n>A trajectory generator first produces human and object trajectories without prescribed waypoints, and an action generator conditions on these paths to synthesize detailed motions.
  arXiv  Detail & Related papers  (2025-12-22T05:33:59Z)
• Kinematify: Open-Vocabulary Synthesis of High-DoF Articulated Objects [59.51185639557874]
  We introduce Kinematify, an automated framework that synthesizes articulated objects directly from arbitrary RGB images or textual descriptions.<n>Our method addresses two core challenges: (i) inferring kinematic topologies for high-DoF objects and (ii) estimating joint parameters from static geometry.
  arXiv  Detail & Related papers  (2025-11-03T07:21:42Z)
• Ego-Vision World Model for Humanoid Contact Planning [17.12603623353369]
  We propose a framework combining a learned world model with sampling-based Model Predictive Control.<n>Our system achieves robust, real-time contact planning from proprioception and ego-centric depth images.
  arXiv  Detail & Related papers  (2025-10-13T17:47:39Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback [8.420670642409219]
  We propose a representation that jointly models object deformations and contact patches from visuo-tactile feedback. We propose a neural network architecture to learn a NDCF, and train it using simulated data. We demonstrate that the learned NDCF transfers directly to the real-world without the need for fine-tuning.
  arXiv  Detail & Related papers  (2023-05-23T18:53:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.