ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions

• URL: http://arxiv.org/abs/2607.11570v1
• Date: Mon, 13 Jul 2026 13:50:43 GMT
• Title: ERR@HRI 3.0 Challenge: Multimodal Detection of Errors and Anticipation in Human-Robot Interactions
• Abstract summary: The third edition of the ERR@HRI Challenge (ERR@HRI 3.0) provided researchers with two complementary datasets.<n>This paper describes the datasets, tasks, baselines, and results of ERR@HRI 3.0, and discusses implications for building generalizable, context-aware, and anticipatory error detection systems.
• Score: 18.151307410451796
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As robots become increasingly integrated into human environments, their ability to detect and respond to errors remains critical for maintaining user trust and interaction quality. While recent advances in machine learning have improved error detection capabilities, most approaches are limited to specific contexts, controlled settings, or pre-extracted features, limiting their generalizability and applicability to real-world conditions. To address this challenge, the third edition of the ERR@HRI Challenge (ERR@HRI 3.0) provided researchers with two complementary datasets that enable end-to-end innovation in methods for both detecting and preventing errors in human-robot interaction. The challenge offered raw, non-anonymized video data from naturalistic settings: (1) the Bystander Affect Detection (BAD) dataset, containing webcam recordings of 45 participants' spontaneous reactions to robot and human failure scenarios; and (2) the Bad Idea dataset, featuring 29 participants' anticipatory facial responses while predicting action outcomes before failures occur. Both datasets were collected via crowdsourcing, capturing the inherent variability of real-world conditions. This naturalistic variability, while challenging, provides an authentic testbed for developing robust error detection systems. Participants developed multimodal machine learning models for bystander reaction detection (Track 1) and anticipatory outcome prediction (Track 2), with an optional cross-dataset generalization track (Track 3). Three teams submitted valid models, all of which surpassed our convolutional neural network baselines. This paper describes the datasets, tasks, baselines, and results of ERR@HRI 3.0, and discusses implications for building generalizable, context-aware, and anticipatory error detection systems for human-robot interaction.

Related papers

• RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction [51.76737991200891]
  Human-to-robot (H2R) object handover is a fundamental capability for human-robot collaboration, yet progress is hindered by the scarcity of large-scale, human-centric datasets.<n>We introduce Hand2Bot, an RGB-D video dataset that provides rich contextual information such as body posture and facial expressions.<n>We propose PassGen, a generative pipeline that synthesizes realistic handover sequences while ensuring hand-object consistency.
  arXiv  Detail & Related papers  (2026-08-13T09:55:59Z)
• Myopia Prevention and Control 3.0: Artificial Intelligence--Driven Risk Stratification, Proactive Monitoring, and Personalized Intervention [1.0582793992259394]
  We review the emergence of Myopia Prevention and Control 3.0, defined by AI integration across three interconnected domains.<n>We critically evaluate evidence across each stage, discuss challenges in data quality, model validation, ethics, and equity.<n>We outline future directions including multimodal foundation models, digital twins, and causal machine learning.
  arXiv  Detail & Related papers  (2026-07-27T09:07:07Z)
• Multimodal Anomaly Detection for Human-Robot Interaction [0.0]
  We propose a framework that transforms video streams into semantically meaningful feature vectors before performing reconstruction-based anomaly detection.<n>We augment these visual feature vectors with the robot's internal sensors' readings and a Scene Graph, enabling the model to capture both external anomalies in the visual environment and internal failures within the robot itself.
  arXiv  Detail & Related papers  (2026-04-10T13:51:19Z)
• RFM-HRI : A Multimodal Dataset of Medical Robot Failure, User Reaction and Recovery Preferences for Item Retrieval Tasks [0.0]
  This work contributes (1) a publicly available multimodal dataset (RFM-HRI), (2) analysis of user responses to different failure types and preferred recovery strategies, and (3) a crash-cart retrieval scenario enabling systematic comparison of recovery strategies with implications for safety-critical failure recovery.
  arXiv  Detail & Related papers  (2026-03-05T19:52:19Z)
• Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models [53.20969621498248]
  We propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures.<n>We construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail.<n>We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection.
  arXiv  Detail & Related papers  (2025-12-01T17:57:27Z)
• HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario [63.77482302352545]
  HHI-Assist is a dataset comprising motion capture clips of human-human interactions in assistive tasks.<n>Our work has the potential to significantly enhance robotic assistance policies.
  arXiv  Detail & Related papers  (2025-09-12T09:38:17Z)
• Bridging the Gap Between Ideal and Real-world Evaluation: Benchmarking AI-Generated Image Detection in Challenging Scenarios [54.07895223545793]
  This paper introduces the Real-World Robustness dataset (RRDataset) for comprehensive evaluation of detection models across three dimensions.<n>RRDataset includes high-quality images from seven major scenarios.<n>We benchmarked 17 detectors and 10 vision-language models (VLMs) on RRDataset and conducted a large-scale human study.
  arXiv  Detail & Related papers  (2025-09-11T06:15:52Z)
• ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations [18.151307410451796]
  ERR@HRI 2.0 Challenge provides a dataset of conversational robot failures during human-robot conversations.<n> dataset includes 16 hours of dyadic human-robot interactions, incorporating facial, speech, and head movement features.<n>Participants are invited to form teams and develop machine learning models that detect these failures using multimodal data.
  arXiv  Detail & Related papers  (2025-07-17T18:21:45Z)
• Towards Unified 3D Object Detection via Algorithm and Data Unification [70.27631528933482]
  We build the first unified multi-modal 3D object detection benchmark MM- Omni3D and extend the aforementioned monocular detector to its multi-modal version. We name the designed monocular and multi-modal detectors as UniMODE and MM-UniMODE, respectively.
  arXiv  Detail & Related papers  (2024-02-28T18:59:31Z)
• Joint Sensing, Communication, and AI: A Trifecta for Resilient THz User Experiences [118.91584633024907]
  A novel joint sensing, communication, and artificial intelligence (AI) framework is proposed so as to optimize extended reality (XR) experiences over terahertz (THz) wireless systems.
  arXiv  Detail & Related papers  (2023-04-29T00:39:50Z)
• Towards Unbiased Visual Emotion Recognition via Causal Intervention [63.74095927462]
  We propose a novel Emotion Recognition Network (IERN) to alleviate the negative effects brought by the dataset bias. A series of designed tests validate the effectiveness of IERN, and experiments on three emotion benchmarks demonstrate that IERN outperforms other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-26T10:40:59Z)
• Few-Shot Visual Grounding for Natural Human-Robot Interaction [0.0]
  We propose a software architecture that segments a target object from a crowded scene, indicated verbally by a human user. At the core of our system, we employ a multi-modal deep neural network for visual grounding. We evaluate the performance of the proposed model on real RGB-D data collected from public scene datasets.
  arXiv  Detail & Related papers  (2021-03-17T15:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.