GenerativeMPC: VLM-RAG-guided Whole-Body MPC with Virtual Impedance for Bimanual Mobile Manipulation

• URL: http://arxiv.org/abs/2604.19522v1
• Date: Tue, 21 Apr 2026 14:44:38 GMT
• Title: GenerativeMPC: VLM-RAG-guided Whole-Body MPC with Virtual Impedance for Bimanual Mobile Manipulation
• Abstract summary: Bimanual mobile manipulation requires a seamless integration between high-level semantic reasoning and safe, compliant physical interaction.<n>This paper presents GenerativeMPC, a hierarchical cyber-physical framework that bridges semantic scene understanding with physical control parameters for bimanual mobile manipulators.
• Score: 2.061143628317803
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bimanual mobile manipulation requires a seamless integration between high-level semantic reasoning and safe, compliant physical interaction - a challenge that end-to-end models approach opaquely and classical controllers lack the context to address. This paper presents GenerativeMPC, a hierarchical cyber-physical framework that explicitly bridges semantic scene understanding with physical control parameters for bimanual mobile manipulators. The system utilizes a Vision-Language Model with Retrieval-Augmented Generation (VLM-RAG) to translate visual and linguistic context into grounded control constraints, specifically outputting dynamic velocity limits and safety margins for a Whole-Body Model Predictive Controller (MPC). Simultaneously, the VLM-RAG module modulates virtual stiffness and damping gains for a unified impedance-admittance controller, enabling context-aware compliance during human-robot interaction. Our framework leverages an experience-driven vector database to ensure consistent parameter grounding without retraining. Experimental results in MuJoCo, IsaacSim, and on a physical bimanual platform confirm a 60% speed reduction near humans and safe, socially-aware navigation and manipulation through semantic-to-physical parameter grounding. This work advances the field of human-centric cybernetics by grounding large-scale cognitive models into predictable, high-frequency physical control loops.

Related papers

• WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation [85.15806404962193]
  Action-conditioned world models (ACWMs) promise to provide embodied AI with scalable predictive simulators for planning, policy evaluation, and data generation.<n>We evaluate ACWMs through the observable capabilities expected of physical simulators.<n>WorldSimProbe reveals systematic action-realization degradation across control variation, structured failures in interaction grounding and dynamics.
  arXiv  Detail & Related papers  (2026-08-10T08:48:06Z)
• In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics [109.99871326497333]
  In-Context Model Predictive Generation (ICMPG) is a framework that integrates language-model planning with inference-time physical feedback.<n>ICMPG reformulates motion synthesis as a Model Predictive Control (MPC)-like process with two modules.
  arXiv  Detail & Related papers  (2026-06-25T12:50:33Z)
• SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-based Humanoid Control [72.5651722107621]
  SCRIPT is a scalable diffusion policy with a multi-stage training framework for language-driven physics-based humanoid control.<n>The core of SCRIPT is a Joint Action-State-Text Diffusion Transformer (JAST-DiT), which represents actions, physical states, and text as dedicated token streams.<n>To stabilize autore control, we introduce a nonlinear history conditioning mechanism, which preserves the dense recent context and samples increasingly sparse cues from long-term history.
  arXiv  Detail & Related papers  (2026-05-21T14:17:21Z)
• HybridMimic: Hybrid RL-Centroidal Control for Humanoid Motion Mimicking [3.446278503007937]
  HybridMimic is a framework in which a learned policy dynamically modulates a centroidal-model-based controller.<n>It reduces the average base position tracking error by 13% compared to a state-of-the-art RL baseline.
  arXiv  Detail & Related papers  (2026-03-06T17:58:02Z)
• PMG: Parameterized Motion Generator for Human-like Locomotion Control [14.637220434597168]
  We develop a real-time motion generator that produces human-like locomotion in a single integrated system.<n>We show that, within a single integrated system, PMG produces natural, human-like locomotion, responds precisely to high-dimensional control inputs.<n>These results establish a practical, experimentally validated pathway toward natural and deployable humanoid control.
  arXiv  Detail & Related papers  (2026-02-13T06:38:04Z)
• HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation [2.460614761319643]
  This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate impedance parameters and gripper configurations directly from an egocentric RGB image.<n>We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%.
  arXiv  Detail & Related papers  (2026-01-21T11:04:19Z)
• mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs [5.109732854501585]
  We introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations.<n>Our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.
  arXiv  Detail & Related papers  (2025-12-17T18:47:31Z)
• Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation [70.8381970762877]
  Vision-Language Models (VLMs) have demonstrated remarkable capabilities in semantic reasoning and task planning.<n>We introduce GRACE, a novel framework that grounds VLM-based reasoning through executable analytic concepts.<n>G GRACE provides a unified and interpretable interface between high-level instruction understanding and low-level robot control.
  arXiv  Detail & Related papers  (2025-10-09T09:08:33Z)
• MetAdv: A Unified and Interactive Adversarial Testing Platform for Autonomous Driving [85.04826012938642]
  MetAdv is a novel adversarial testing platform that enables realistic, dynamic, and interactive evaluation.<n>It supports flexible 3D vehicle modeling and seamless transitions between simulated and physical environments.<n>It enables real-time capture of physiological signals and behavioral feedback from drivers.
  arXiv  Detail & Related papers  (2025-08-04T03:07:54Z)
• TLControl: Trajectory and Language Control for Human Motion Synthesis [68.09806223962323]
  We present TLControl, a novel method for realistic human motion synthesis. It incorporates both low-level Trajectory and high-level Language semantics controls. It is practical for interactive and high-quality animation generation.
  arXiv  Detail & Related papers  (2023-11-28T18:54:16Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.