HumanCLAW: Can Vision-Language Models Act Through a Body?
Abstract Overview
HumanCLAW is an evaluation framework for testing whether vision-language models can make effective embodied decisions when their outputs are executed through a humanoid body. The framework separates high-level action choice from low-level motor execution by having a frozen, off-the-shelf VLM issue atomic skill commands, which are then converted into short full-body motions and executed in a half-physics simulator with gravity, collisions, and object interactions. On top of this setup, the authors build HumanCLAW-Bench, a benchmark of 1,218 long-horizon egocentric find-navigate-interact episodes across 41 indoor scenes. The evaluation is designed to attribute failure primarily to decision-making rather than locomotion instability, and it measures progressive task success, motion coherence, body awareness, disturbance, and cost. Across the tested models, the main limitation is not target recognition once visible, but maintaining awareness of the body's state and its relation to the environment during navigation and interaction.
Novelty
The paper’s main novelty is an evaluation setting that keeps embodied action physically grounded while explicitly factoring out low-level control failures, allowing assessment of a VLM’s decision quality through a body rather than its motor policy. It also introduces a benchmark and skill-based execution interface that connect frozen general-purpose VLMs to continuous humanoid motion without retraining the decision maker.
Results
In experiments on nine state-of-the-art VLMs, none solved the benchmark reliably; the best reported model, Gemini-3.1, achieved 16.8% interaction success and 42.4% navigation success. The study finds that recognizing targets is comparatively less problematic than embodied self-awareness: models often fail to judge whether they have arrived, whether they are blocked, or how to place the body correctly for sitting. Ablations further show that the skill-specific verifier is critical for successful closed-loop behavior, while longer text history offers limited gains and too many visual-history frames can degrade performance.
Key Points
- HumanCLAW decouples embodied action decision-making from low-level execution by mapping VLM-issued atomic skills to short full-body motions executed in a half-physics simulator.
- HumanCLAW-Bench contains 1,218 long-horizon egocentric episodes across 41 indoor scenes and evaluates staged success together with motion quality, collisions, disturbance, and computational cost.
- Across nine VLMs, the dominant bottleneck is embodied self-awareness rather than recognition: models often see the target but fail to localize their body relative to it or execute the final interaction correctly.
References
- arXiv: https://arxiv.org/abs/2607.27180v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2607.27180v1
- Hugging Face Papers: https://huggingface.co/papers/2607.27180
- Project: https://human-claw.github.io