Probabilistic Human Intent Prediction for Mobile Manipulation: An Evaluation with Human-Inspired Constraints

• URL: http://arxiv.org/abs/2507.10131v1
• Date: Mon, 14 Jul 2025 10:21:27 GMT
• Title: Probabilistic Human Intent Prediction for Mobile Manipulation: An Evaluation with Human-Inspired Constraints
• Abstract summary: Accurate inference of human intent enables human-robot collaboration without constraining human control or causing conflicts between humans and robots.<n>We present GUIDER, a probabilistic framework that enables a robot to estimate the intent of human operators.<n>We evaluated GUIDER on 25 trials (five participants x five task variants) in Isaac Sim, and compared it with two baselines, one for navigation and one for manipulation.
• Score: 2.2893865000399938
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate inference of human intent enables human-robot collaboration without constraining human control or causing conflicts between humans and robots. We present GUIDER (Global User Intent Dual-phase Estimation for Robots), a probabilistic framework that enables a robot to estimate the intent of human operators. GUIDER maintains two coupled belief layers, one tracking navigation goals and the other manipulation goals. In the Navigation phase, a Synergy Map blends controller velocity with an occupancy grid to rank interaction areas. Upon arrival at a goal, an autonomous multi-view scan builds a local 3D cloud. The Manipulation phase combines U2Net saliency, FastSAM instance saliency, and three geometric grasp-feasibility tests, with an end-effector kinematics-aware update rule that evolves object probabilities in real-time. GUIDER can recognize areas and objects of intent without predefined goals. We evaluated GUIDER on 25 trials (five participants x five task variants) in Isaac Sim, and compared it with two baselines, one for navigation and one for manipulation. Across the 25 trials, GUIDER achieved a median stability of 93-100% during navigation, compared with 60-100% for the BOIR baseline, with an improvement of 39.5% in a redirection scenario (T5). During manipulation, stability reached 94-100% (versus 69-100% for Trajectron), with a 31.4% difference in a redirection task (T3). In geometry-constrained trials (manipulation), GUIDER recognized the object intent three times earlier than Trajectron (median remaining time to confident prediction 23.6 s vs 7.8 s). These results validate our dual-phase framework and show improvements in intent inference in both phases of mobile manipulation tasks.

Related papers

• HuRo: Robotizing Human Videos for Scalable VLA Pretraining [21.648183246162237]
  We develop a robotization pipeline that converts heterogeneous human videos into robot-aligned observations and action trajectories.<n>We construct the HuRo dataset, comprising about 630K robotized episodes and 142M processed frames from five human-video sources.
  arXiv  Detail & Related papers  (2026-09-09T18:02:05Z)
• GigaBrain-WBC-0.5: A Behavior World Model for Robust Whole-Body Control with Environment Interaction [68.67772166423812]
  We present GigaBrain-WBC-0.5, the first Behavior World Model (BWM) for humanoid whole-body control.<n>Rather than a purely reactive tracker, we train a causal Transformer to jointly predict its next action, next state, and the distribution over its next latent behavior command.<n>An automatic terrain-annotation pipeline recovers full 3D contact geometry from retargeted motion, enabling terrain annotation at the scale of existing motion datasets.
  arXiv  Detail & Related papers  (2026-08-18T18:21:36Z)
• GUIDER: Evaluating Goal-Free Human Intent Inference for Teleoperated Manipulation on Real-Robot Data [1.7107261936393963]
  We deploy the Global User Intent Dual-phase Estimation for Robots to test the manipulation phase across various assistance scenarios.<n>We add online probability updates, workspace limits, support-plane filtering, and a grasping mode that prioritizes feasible grasp regions.<n>Across 20 manipulation steps in three scenarios, GUIDER estimated human intent within the correct grasp-candidate set in all cases.
  arXiv  Detail & Related papers  (2026-08-15T23:16:24Z)
• WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation [49.31386176285509]
  WatchAct is a benchmark for robot manipulation grounded in observed human behavior.<n>WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains.<n>In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct.
  arXiv  Detail & Related papers  (2026-06-24T23:13:56Z)
• Forecast-aware Gaussian Splatting for Predictive 3D Representation in Language-Guided Pick-and-Place Manipulation [4.711302998453154]
  We introduce Forecast-aware Gaussian Splatting (Forecast-GS), a predictive 3D representation framework for language-conditioned robotic manipulation.<n>We validate Forecast-GS on real-world pick-and-place manipulation tasks, including Cutter-to-Box, Apple-to-Bowl, and Sponge-to-Tray.<n>Forecast-GS with automatic candidate selection achieves success rates of 21/25, 23/25, and 16/25 on the three tasks, respectively.
  arXiv  Detail & Related papers  (2026-05-11T18:48:26Z)
• Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics [0.0]
  We present an offline hand-shadowing and mapping pipeline from a single egocentric RGB-D camera mounted on 3D-printed glasses.<n>The pipeline detects 21 hand landmarks per hand using MediaPipe Hands.<n>A gripper controller maps thumb-index finger geometry to grasp aperture with a four-level fallback hierarchy.
  arXiv  Detail & Related papers  (2026-03-11T23:53:28Z)
• Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation [14.833622989644352]
  We develop a lightweight, fully on-device pipeline that converts natural-language commands into reliable manipulation.<n>Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint.<n>Results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and vision-guided grounding, provides a practical path to deterministic, real-time manipulation.
  arXiv  Detail & Related papers  (2026-02-10T16:25:39Z)
• UPTor: Unified 3D Human Pose Dynamics and Trajectory Prediction for Human-Robot Interaction [0.688204255655161]
  We propose a technique to predict full-body pose and trajectory key-points in a global coordinate frame.<n>We use an off-the-shelf 3D human pose estimation module, a graph attention network, and a compact, non-autoregressive transformer.<n>In comparison to prior work, we show that our approach is compact, real-time, and accurate in predicting human navigation motion across all datasets.
  arXiv  Detail & Related papers  (2025-05-20T19:57:25Z)
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [51.49400490437258]
  This work develops a method for imitating articulated object manipulation from a single monocular RGB human demonstration. We first propose 4D Differentiable Part Models (4D-DPM), a method for recovering 3D part motion from a monocular video. Given this 4D reconstruction, the robot replicates object trajectories by planning bimanual arm motions that induce the demonstrated object part motion. We evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part trajectories and RSRD's physical execution performance on 9 objects across 10 trials each on a bimanual YuMi robot.
  arXiv  Detail & Related papers  (2024-09-26T17:57:16Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation [65.46610405509338]
  We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We show that this approach of combining scalably learned track prediction with a residual policy enables diverse generalizable robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)
• RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation [20.685168082722534]
  We present RiEMann, an end-to-end near Real-time SE(3)-Equivariant Robot Manipulation imitation learning framework from scene point cloud input. Compared to previous methods that rely on descriptor field matching, RiEMann directly predicts the target poses of objects for manipulation without any object segmentation. RiEMann learns a manipulation task from scratch with 5 to 10 demonstrations, generalizes to unseen SE(3) transformations and instances of target objects, resists visual interference of distracting objects, and follows the near real-time pose change of the target object.
  arXiv  Detail & Related papers  (2024-03-28T14:31:10Z)
• Exploring 3D Human Pose Estimation and Forecasting from the Robot's Perspective: The HARPER Dataset [52.22758311559]
  We introduce HARPER, a novel dataset for 3D body pose estimation and forecast in dyadic interactions between users and Spot. The key-novelty is the focus on the robot's perspective, i.e., on the data captured by the robot's sensors. The scenario underlying HARPER includes 15 actions, of which 10 involve physical contact between the robot and users.
  arXiv  Detail & Related papers  (2024-03-21T14:53:50Z)
• Towards a large-scale fused and labeled dataset of human pose while interacting with robots in shared urban areas [0.0]
  Human Pose Estimation is a vital stepping stone for various downstream applications. We bridge this gap by repurposing, fusing, and labeling two datasets. The resulting unique dataset represents thousands of real-world indoor and outdoor human-robot interaction scenarios.
  arXiv  Detail & Related papers  (2024-01-28T18:24:34Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior.<n>Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)
• Walking = Traversable? : Traversability Prediction via Multiple Human Object Tracking under Occlusion [1.9580473532948401]
  This study presents an innovative approach that replaces first-person-view sensors with a third-person-view monocular camera mounted on the observer robot. The key idea is to use two types of trackers, SLAM and MOT, to monitor stationary objects and moving humans and assess their interactions.
  arXiv  Detail & Related papers  (2023-09-30T03:44:48Z)
• Robots of the Lost Arc: Self-Supervised Learning to Dynamically Manipulate Fixed-Endpoint Cables [31.103646278125318]
  High-speed robot arm motions can manipulate cables to vault over obstacles, knock objects from pedestals, and weave between obstacles. We propose a self-supervised learning framework that enables a UR5 robot to perform these three tasks.
  arXiv  Detail & Related papers  (2020-11-10T00:19:50Z)
• Sim-to-Real Transfer for Vision-and-Language Navigation [70.86250473583354]
  We study the problem of releasing a robot in a previously unseen environment, and having it follow unconstrained natural language navigation instructions. Recent work on the task of Vision-and-Language Navigation (VLN) has achieved significant progress in simulation. To assess the implications of this work for robotics, we transfer a VLN agent trained in simulation to a physical robot.
  arXiv  Detail & Related papers  (2020-11-07T16:49:04Z)
• It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction [59.027152973975575]
  We present Predicted Conditioned Network (PECNet) for flexible human trajectory prediction. PECNet infers distant endpoints to assist in long-range multi-modal trajectory prediction. We show that PECNet improves state-of-the-art performance on the Stanford Drone trajectory prediction benchmark by 20.9% and on the ETH/UCY benchmark by 40.8%.
  arXiv  Detail & Related papers  (2020-04-04T21:27:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.