Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

• URL: http://arxiv.org/abs/2609.07394v1
• Date: Mon, 07 Sep 2026 12:06:48 GMT
• Title: Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities
• Abstract summary: We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view.<n>We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks.
• Score: 27.512271990478737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction [23.90153863744227]
  We introduce The Imitator Game, a four-level benchmark for learning from human video.<n>It progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice.<n>We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date, and Imitator Arena, an open platform for blind A/B human evaluation.
  arXiv  Detail & Related papers  (2026-08-23T08:51:01Z)
• Developing Neural Network-Based Gaze Control Systems for Social Robots [0.0]
  This study aims to develop an empirical motion-time pattern for human gaze behavior in various social situations.<n>Deep learning models, specifically Long Short-Term Memory (LSTM) and Transformers, were used to analyze and predict gaze patterns.
  arXiv  Detail & Related papers  (2026-02-11T15:30:39Z)
• Contour Integration Underlies Human-Like Vision [2.6716072974490794]
  Humans perform at high accuracy, even with few object contours present.<n>Humans exhibit an integration bias -- a preference towards recognizing objects made up of directional fragments over directionless fragments.
  arXiv  Detail & Related papers  (2025-04-07T16:45:06Z)
• Evaluating Multiview Object Consistency in Humans and Image Models [68.36073530804296]
  We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape. We collect 35K trials of behavioral data from over 500 participants. We then evaluate the performance of common vision models.
  arXiv  Detail & Related papers  (2024-09-09T17:59:13Z)
• Sapiens: Foundation for Human Vision Models [14.72839332332364]
  We present Sapiens, a family of models for four fundamental human-centric vision tasks. Our models support 1K high-resolution inference and are easy to adapt for individual tasks. We observe that self-supervised pretraining on a curated dataset of human images significantly boosts the performance for a diverse set of human-centric tasks.
  arXiv  Detail & Related papers  (2024-08-22T17:37:27Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior.<n>Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)
• What's "up" with vision-language models? Investigating their struggle with spatial reasoning [76.2406963762722]
  Three new corpora quantify model comprehension of basic spatial relations. We evaluate 18 vision-language (VL) models, finding that all perform poorly. We conclude by studying causes of this surprising behavior.
  arXiv  Detail & Related papers  (2023-10-30T17:50:15Z)
• Model Predictive Control for Fluid Human-to-Robot Handovers [50.72520769938633]
  Planning motions that take human comfort into account is not a part of the human-robot handover process. We propose to generate smooth motions via an efficient model-predictive control framework. We conduct human-to-robot handover experiments on a diverse set of objects with several users.
  arXiv  Detail & Related papers  (2022-03-31T23:08:20Z)
• Partial success in closing the gap between human and machine vision [30.78663978510427]
  A few years ago, the first CNN surpassed human performance on ImageNet. Here we ask: Are we making progress in closing the gap between human and machine vision? We tested human observers on a broad range of out-of-distribution (OOD) datasets.
  arXiv  Detail & Related papers  (2021-06-14T13:23:35Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)
• LESS is More: Rethinking Probabilistic Models of Human Behavior [36.020541093946925]
  Boltzmann noisily-rational decision model assumes people approximately optimize a reward function. Human trajectories lie in a continuous space, with continuous-valued features that influence the reward function. We introduce a model that explicitly accounts for distances between trajectories, rather than only their rewards.
  arXiv  Detail & Related papers  (2020-01-13T18:59:01Z)
• Learning Predictive Models From Observation and Interaction [137.77887825854768]
  Learning predictive models from interaction with the world allows an agent, such as a robot, to learn about how the world works. However, learning a model that captures the dynamics of complex skills represents a major challenge. We propose a method to augment the training set with observational data of other agents, such as humans.
  arXiv  Detail & Related papers  (2019-12-30T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.