Face versus Body Tracking for Human-Robot Interaction: An Egocentric Dataset

• URL: http://arxiv.org/abs/2606.03694v1
• Date: Tue, 02 Jun 2026 14:15:17 GMT
• Title: Face versus Body Tracking for Human-Robot Interaction: An Egocentric Dataset
• Abstract summary: State-of-the-art computer vision models are heavily optimized for surveillance or autonomous driving.<n>A social robot faces distinct egocentric challenges, such as humans bouncing, obstructing each other, or leaving the frame.<n>We introduce a novel, custom-annotated egocentric dataset collected via the Furhat robot to capture complex social dynamics.
• Score: 7.3852311170912905
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To enable meaningful human-robot interaction (HRI), a robot must continuously assess engagement by consistently tracking users over time. State-of-the-art computer vision models, however, are heavily optimized for surveillance or autonomous driving. A social robot faces distinct egocentric challenges, such as humans bouncing, obstructing each other, or leaving the frame. Frequent identity switches (IDSW) cause the robot to lose its footing mid-conversation. To address this, we introduce a novel, custom-annotated egocentric dataset collected via the Furhat robot to capture complex social dynamics. We present a systematic evaluation isolating detection errors from tracking logic, comparing face versus body tracking, and assessing the impact of extended spatial memory and appearance re-identification (ReID). Results indicate that increasing spatial memory mitigates prolonged occlusions but fails on complex dynamic events. Integrating ReID resolves complex switches but exhibits opposing effects: it substantially improves body tracking stability, yet causes facial IDSW to spike due to profile angle sensitivity. Ultimately, our optimized pipeline reduces IDSW by 49\%, mitigating interaction breakdowns. Because standard benchmarks lack dense, close-quarter occlusions, this work highlights the critical need for natively captured social dynamics to truly validate HRI perception models.

Related papers

• SocioGesture: Real-Time and Adaptive Social Gesture Perception for Human-Robot Interaction [4.0450087039946006]
  We present SocioGesture, a real-time adaptive social gesture perception system for human-robot interaction (HRI)<n>SocioGesture uses a compact confidence-aware body-hand skeleton representation and a lightweight dual-stream model that fuses body motion with hand articulation for low-latency onboard recognition.<n>On a social gesture dataset collected in mixed indoor-outdoor HRI scenarios, SocioGesture achieves strong held-out-subject recognition.<n>During deployment, uncertain interaction segments are saved for offline labeling and adaptation, enabling SocioGesture to expand its gesture vocabulary.
  arXiv  Detail & Related papers  (2026-09-03T23:03:43Z)
• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• EgoHTR: Egocentric 4D Demonstrations of Human Terrain Traversal [63.906929035961134]
  We open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments.<n>The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth.<n>Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build context-aware robots.
  arXiv  Detail & Related papers  (2026-07-15T06:02:41Z)
• Seeing Through Occlusion: Deterministic Arm Kinematic Correction for Robot Teleoperation [6.979012378888037]
  This paper presents an Arm Kinematic Correction (AKC) method that improves depth estimation by enforcing geometric constraints based on constant arm lengths.<n> Experimental validation against a Vicon reference system demonstrates reliable performance for both static and dynamic joint motions.<n>The results show that AKC enhances robustness and preserves anatomical consistency under long-duration, severe self-occlusion, even when paired with less reliable temporal filters.
  arXiv  Detail & Related papers  (2026-06-17T16:20:10Z)
• Does Appearance Help? A Systematic Study of Image-Based Re-Identification in Online 3D Multi-Pedestrian Tracking [0.6117371161379209]
  This work presents a systematic study of image-based ReID in online 3D MOT, utilizing a lightweight projection-based framework.<n>Experiments on the Pedestrian class of the KITTI dataset reveal that naive linear fusion, of appearance and motion costs, degrades performance due to visual noise.<n>We show that lightweight architectures can offer an optimal trade-off between the low latency required for safe navigation and the discriminative power needed for social awareness.
  arXiv  Detail & Related papers  (2026-06-05T13:00:04Z)
• SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance [49.69016078147708]
  Accurately reconstructing human behavior in close-interaction scenarios is crucial for enabling realistic virtual interactions in augmented reality.<n>We propose SocialMirror, a diffusion-based framework that integrates semantic and geometric cues to effectively address these issues.<n>SocialMirror achieves state-of-the-art performance in reconstructing interactive human meshes, demonstrating strong generalization across unseen datasets and in-the-wild scenarios.
  arXiv  Detail & Related papers  (2026-04-15T07:41:52Z)
• From Obstacles to Etiquette: Robot Social Navigation with VLM-Informed Path Selection [57.74400052368147]
  This paper presents a social robot navigation framework that integrates geometric planning with contextual social reasoning.<n>The system first extracts obstacles and human dynamics to generate geometrically feasible candidate paths, then leverages a fine-tuned vision-language model (VLM) to evaluate these paths.<n>Experiments in four social navigation contexts demonstrate that our method achieves the best overall performance with the lowest personal space violation duration, the minimal pedestrian-facing time, and no social zone intrusions.
  arXiv  Detail & Related papers  (2026-02-09T18:46:12Z)
• Learning Whole-Body Human-Humanoid Interaction from Human-Human Demonstrations [63.80827184637476]
  We introduce D-STAR, a hierarchical policy that disentangles when to act from where to act.<n>We validate our framework through extensive and rigorous simulations.
  arXiv  Detail & Related papers  (2026-01-14T14:37:06Z)
• Real-Time Human-Robot Interaction Intent Detection Using RGB-based Pose and Emotion Cues with Cross-Camera Model Generalization [0.8839687029212673]
  Service robots in public spaces require real-time understanding of human behavioral intentions for natural interaction.<n>We present a framework for frame-accurate human-robot interaction intent detection that fuses camera-invariant 2D skeletal pose and facial emotion features extracted from monocular RGB video.
  arXiv  Detail & Related papers  (2025-12-18T08:44:22Z)
• HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario [63.77482302352545]
  HHI-Assist is a dataset comprising motion capture clips of human-human interactions in assistive tasks.<n>Our work has the potential to significantly enhance robotic assistance policies.
  arXiv  Detail & Related papers  (2025-09-12T09:38:17Z)
• Siamese Network with Dual Attention for EEG-Driven Social Learning: Bridging the Human-Robot Gap in Long-Tail Autonomous Driving [0.0]
  This study presents a brain-computer interface (BCI) framework that enables classification of Electroencephalogram (EEG) signals to detect cognitively demanding and safety-critical events.<n>As a timely and motivating co-robotic engineering application, we simulate a human-in-the-loop scenario to flag risky events in semi-autonomous robotic driving.<n>We propose a dual-attention Siamese convolutional network paired with Dynamic Time Warping Barycenter Averaging approach to generate robust EEG-encoded signal representations.
  arXiv  Detail & Related papers  (2025-04-14T15:06:17Z)
• IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction [1.3597551064547502]
  Predicting the trajectory of pedestrians in crowd scenarios is indispensable in self-driving or autonomous mobile robot field. Previous researchers focused on how to model human-human interactions but neglected the relative importance of interactions. New mechanism based on correntropy is introduced to measure the relative importance of human-human interactions.
  arXiv  Detail & Related papers  (2023-11-26T05:17:11Z)
• Occlusion-Robust Multi-Sensory Posture Estimation in Physical Human-Robot Interaction [10.063075560468798]
  2D postures from OpenPose over a single camera, and the trajectory of the interacting robot while the human performs a task. We use 2D postures from OpenPose over a single camera, and the trajectory of the interacting robot while the human performs a task. We show that our multi-sensory system resolves human kinematic redundancy better than posture estimation solely using OpenPose or posture estimation solely using the robot's trajectory.
  arXiv  Detail & Related papers  (2022-08-12T20:41:09Z)
• RobustFusion: Robust Volumetric Performance Reconstruction under Human-object Interactions from Monocular RGBD Stream [27.600873320989276]
  High-quality 4D reconstruction of human performance with complex interactions to various objects is essential in real-world scenarios. Recent advances still fail to provide reliable performance reconstruction. We propose RobustFusion, a robust volumetric performance reconstruction system for human-object interaction scenarios.
  arXiv  Detail & Related papers  (2021-04-30T08:41:45Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Online Body Schema Adaptation through Cost-Sensitive Active Learning [63.84207660737483]
  The work was implemented in a simulation environment, using the 7DoF arm of the iCub robot simulator. A cost-sensitive active learning approach is used to select optimal joint configurations. The results show cost-sensitive active learning has similar accuracy to the standard active learning approach, while reducing in about half the executed movement.
  arXiv  Detail & Related papers  (2021-01-26T16:01:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.