HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation

• URL: http://arxiv.org/abs/2601.14874v1
• Date: Wed, 21 Jan 2026 11:04:19 GMT
• Title: HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation
• Abstract summary: This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate impedance parameters and gripper configurations directly from an egocentric RGB image.<n>We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%.
• Score: 2.460614761319643
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Humanoid robots must adapt their contact behavior to diverse objects and tasks, yet most controllers rely on fixed, hand-tuned impedance gains and gripper settings. This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate Cartesian impedance parameters and gripper configurations directly from an egocentric RGB image. The system couples a vision-language model for semantic task inference with a FAISS-based Retrieval-Augmented Generation (RAG) module that retrieves experimentally validated stiffness-damping pairs and object-specific grasp angles from two custom databases, and executes them through a task-space impedance controller for compliant manipulation. We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%. Real-world experiments show stable interaction dynamics, with z-axis tracking errors typically within 1-3.5 cm and virtual forces consistent with task-dependent impedance settings. These results demonstrate the feasibility of linking semantic perception with retrieval-based control as an interpretable path toward adaptive humanoid manipulation.

Related papers

• DreamMimic: Learning Visuomotor Whole-Body Loco-Manipulation via World Model [5.082137967180834]
  Vision-based whole-body loco-manipulation on humanoid robots is challenging due to partial observability, contact-rich dynamics, and the difficulty of learning long-horizon behaviors from high-dimensional visual inputs.<n>We present a framework that distills privileged teacher policies into vision-based humanoid controllers via world-model-assisted distillation.
  arXiv  Detail & Related papers  (2026-08-23T08:08:39Z)
• Scaling Behavior Foundation Model for Humanoid Robots [77.31115920959364]
  Behavior Foundation Models (BFMs) have emerged as a promising solution to address challenges by leveraging large-scale behavioral data.<n>We show that our approach yields significant improvements in control fidelity and task generalization.<n>These results establish BFM as a principled and effective foundation for scalable and general-purpose humanoid control.
  arXiv  Detail & Related papers  (2026-07-16T16:08:27Z)
• Beyond Heuristics: A Standardized Real2Sim Pipeline for Physical Human Robot Interaction in Human-in-the-Loop Simulation [7.749790302832782]
  We present a Real2Sim pipeline that identifies the coupled Physical Human-Robot Interaction (pHRI) dynamics of a pelvis-elastic-strap interface on an overground mobile balance assistant.<n>The pipeline thus improves HITL simulation fidelity and supports the Human Digital Twin as a predictive tool for pre-clinical verification of personalized controllers.
  arXiv  Detail & Related papers  (2026-07-03T06:52:37Z)
• HALOMI: Learning Humanoid Loco-Manipulation with Active Perception from Human Demonstrations [10.378782873235453]
  We present HALOMI, a framework for learning humanoid loco-manipulation with active perception from human demonstrations.<n>We propose a manifold-constrained controller that plans in a learned latent behavior manifold to enable precise and robust head-hand tracking in the world frame.<n>We validate HALOMI on a Unitree G1 humanoid robot with an actuated neck across five real-world tasks involving navigation, grasping, bimanual manipulation, whole-body coordination, and dynamic behaviors.
  arXiv  Detail & Related papers  (2026-06-17T07:33:37Z)
• GenerativeMPC: VLM-RAG-guided Whole-Body MPC with Virtual Impedance for Bimanual Mobile Manipulation [2.061143628317803]
  Bimanual mobile manipulation requires a seamless integration between high-level semantic reasoning and safe, compliant physical interaction.<n>This paper presents GenerativeMPC, a hierarchical cyber-physical framework that bridges semantic scene understanding with physical control parameters for bimanual mobile manipulators.
  arXiv  Detail & Related papers  (2026-04-21T14:44:38Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation [55.467742403416175]
  We introduce a physics-driven neural algorithm that translates large-scale motion capture to humanoid embodiments.<n>We learn a unified multimodal controller that supports both dense references and sparse task specifications.<n>Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception.
  arXiv  Detail & Related papers  (2026-03-03T18:59:29Z)
• Generalizable Geometric Prior and Recurrent Spiking Feature Learning for Humanoid Robot Manipulation [90.90219129619344]
  This paper presents a novel R-prior-S, Recurrent Geometric-priormodal Policy with Spiking features.<n>To ground high-level reasoning in physical reality, we leverage lightweight 2D geometric inductive biases.<n>For the data efficiency issue in robotic action generation, we introduce a Recursive Adaptive Spiking Network.
  arXiv  Detail & Related papers  (2026-01-13T23:36:30Z)
• TrajBooster: Boosting Humanoid Whole-Body Manipulation via Trajectory-Centric Learning [79.59753528758361]
  We present TrajBooster, a cross-embodiment framework that leverages abundant wheeled-humanoid data to boost bipedal VLA.<n>Our key idea is to use end-effector trajectories as a morphology-agnostic interface.<n>Results show that TrajBooster allows existing wheeled-humanoid data to efficiently strengthen bipedal humanoid VLA performance.
  arXiv  Detail & Related papers  (2025-09-15T12:25:39Z)
• Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integration [26.351720551267846]
  Tactile and kinesthetic perceptions are crucial for human dexterous manipulation, enabling reliable grasping of objects via sensorimotor integration.<n>We propose a novel glove-mediated tactile-kinematic perception-prediction framework for grasp skill transfer from human intuitive and natural operation to robotic execution based on imitation learning.
  arXiv  Detail & Related papers  (2025-09-10T07:44:12Z)
• Humanoid-VLA: Towards Universal Humanoid Control with Visual Integration [28.825612240280822]
  We propose a novel framework that integrates language understanding, egocentric scene perception, and motion control, enabling universal humanoid control.<n>Humanoid-VLA begins with language-motion pre-alignment using non-egocentric human motion datasets paired with textual descriptions.<n>We then incorporate egocentric visual context through a parameter efficient video-conditioned fine-tuning, enabling context-aware motion generation.
  arXiv  Detail & Related papers  (2025-02-20T18:17:11Z)
• Towards a Unified Transformer-based Framework for Scene Graph Generation and Human-object Interaction Detection [116.21529970404653]
  We introduce SG2HOI+, a unified one-step model based on the Transformer architecture. Our approach employs two interactive hierarchical Transformers to seamlessly unify the tasks of SGG and HOI detection. Our approach achieves competitive performance when compared to state-of-the-art HOI methods.
  arXiv  Detail & Related papers  (2023-11-03T07:25:57Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Learning Compliance Adaptation in Contact-Rich Manipulation [81.40695846555955]
  We propose a novel approach for learning predictive models of force profiles required for contact-rich tasks. The approach combines an anomaly detection based on Bidirectional Gated Recurrent Units (Bi-GRU) and an adaptive force/impedance controller.
  arXiv  Detail & Related papers  (2020-05-01T05:23:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.