Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration

• URL: http://arxiv.org/abs/2605.31196v1
• Date: Fri, 29 May 2026 12:04:38 GMT
• Title: Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration
• Abstract summary: We introduce TouchSafeBench, a benchmark for evaluating collision grounding in vision-language models (VLMs)<n>TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social navigation and social rearrangement.<n>We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact.
• Score: 19.192199380267393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safe human--robot collaboration requires more than visual description: a monitor must determine whether the robot body is safely separated, already colliding with the scene or a person, or about to collide. We call this capability collision grounding: binding visual observations to robot body geometry, camera viewpoint, scene layout, human proximity, and temporal motion in order to infer present and imminent contact. We introduce TouchSafeBench, a physics-grounded benchmark for evaluating collision grounding in vision-language models (VLMs). Built in Habitat~3.0, TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social navigation and social rearrangement, with synchronized multi-view RGB-D observations, top-down trajectory maps, calibrated camera metadata, and simulator-derived contact labels. We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact. Across three frontier or robotics-oriented VLMs and nine visual representations, current models remain far from reliable: the best average Macro-F1 stays below 50\%, explicit depth is not automatically transformed into robot-body collision evidence, and robot--scene contact is consistently harder than human-contact risk. TouchSafeBench reveals a central limitation of embodied VLMs: visual fluency does not imply physical accountability. Reliable robot safety monitors will need representations that explicitly bind viewpoint, robot morphology, metric geometry, and future collision. We will release the benchmark upon acceptance.

Related papers

• Collision-Aware Humanoid Whole-Body Control under Imperfect Tracking Targets [11.010074477825272]
  Whole-body controllers (WBCs) execute commands that track targets while maintaining balance and stability.<n>Most WBCs are blind to scene geometry, which can lead to collisions due to perception, planning, or teleoperation errors.<n>We propose RECAL, a layer that wraps a blind WBC to trade off target tracking against collision avoidance using external scene geometry.
  arXiv  Detail & Related papers  (2026-09-14T22:22:02Z)
• RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction [33.04319504310729]
  RoboGesture is a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system.<n>We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale audio-motion pairs.<n>Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-27T02:13:46Z)
• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training [20.414478780328437]
  Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay.<n>We present textitGenHOI, a framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner.<n>We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks.
  arXiv  Detail & Related papers  (2026-06-11T07:31:05Z)
• OmniRobotHome: A Multi-Camera Platform for Real-Time Multiadic Human-Robot Interaction [21.78933851010173]
  We present OmniRobotHome, the first room-scale residential platform that unifies real-time 3D human and object perception with coordinated multi-robot actuation.<n>We focus on two central problems: safety in shared human-robot environments and human-anticipatory robotic assistance.
  arXiv  Detail & Related papers  (2026-04-30T17:59:58Z)
• EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents [85.77432303199176]
  We propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones.<n>Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes.<n>Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via
  arXiv  Detail & Related papers  (2026-02-26T16:53:41Z)
• Real-Time Human-Robot Interaction Intent Detection Using RGB-based Pose and Emotion Cues with Cross-Camera Model Generalization [0.8839687029212673]
  Service robots in public spaces require real-time understanding of human behavioral intentions for natural interaction.<n>We present a framework for frame-accurate human-robot interaction intent detection that fuses camera-invariant 2D skeletal pose and facial emotion features extracted from monocular RGB video.
  arXiv  Detail & Related papers  (2025-12-18T08:44:22Z)
• HOSIG: Full-Body Human-Object-Scene Interaction Generation with Hierarchical Scene Perception [57.37135310143126]
  HO SIG is a novel framework for synthesizing full-body interactions through hierarchical scene perception.<n>Our framework supports unlimited motion length through autoregressive generation and requires minimal manual intervention.<n>This work bridges the critical gap between scene-aware navigation and dexterous object manipulation.
  arXiv  Detail & Related papers  (2025-06-02T12:08:08Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [78.14584238127338]
  We evaluate single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation.<n>Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• Exploring 3D Human Pose Estimation and Forecasting from the Robot's Perspective: The HARPER Dataset [52.22758311559]
  We introduce HARPER, a novel dataset for 3D body pose estimation and forecast in dyadic interactions between users and Spot. The key-novelty is the focus on the robot's perspective, i.e., on the data captured by the robot's sensors. The scenario underlying HARPER includes 15 actions, of which 10 involve physical contact between the robot and users.
  arXiv  Detail & Related papers  (2024-03-21T14:53:50Z)
• COPILOT: Human-Environment Collision Prediction and Localization from Egocentric Videos [62.34712951567793]
  The ability to forecast human-environment collisions from egocentric observations is vital to enable collision avoidance in applications such as VR, AR, and wearable assistive robotics. We introduce the challenging problem of predicting collisions in diverse environments from multi-view egocentric videos captured from body-mounted cameras. We propose a transformer-based model called COPILOT to perform collision prediction and localization simultaneously.
  arXiv  Detail & Related papers  (2022-10-04T17:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.