Multimodal Anomaly Detection for Human-Robot Interaction

• URL: http://arxiv.org/abs/2604.09326v1
• Date: Fri, 10 Apr 2026 13:51:19 GMT
• Title: Multimodal Anomaly Detection for Human-Robot Interaction
• Abstract summary: We propose a framework that transforms video streams into semantically meaningful feature vectors before performing reconstruction-based anomaly detection.<n>We augment these visual feature vectors with the robot's internal sensors' readings and a Scene Graph, enabling the model to capture both external anomalies in the visual environment and internal failures within the robot itself.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring safety and reliability in human-robot interaction (HRI) requires the timely detection of unexpected events that could lead to system failures or unsafe behaviours. Anomaly detection thus plays a critical role in enabling robots to recognize and respond to deviations from normal operation during collaborative tasks. While reconstruction models have been actively explored in HRI, approaches that operate directly on feature vectors remain largely unexplored. In this work, we propose MADRI, a framework that first transforms video streams into semantically meaningful feature vectors before performing reconstruction-based anomaly detection. Additionally, we augment these visual feature vectors with the robot's internal sensors' readings and a Scene Graph, enabling the model to capture both external anomalies in the visual environment and internal failures within the robot itself. To evaluate our approach, we collected a custom dataset consisting of a simple pick-and-place robotic task under normal and anomalous conditions. Experimental results demonstrate that reconstruction on vision-based feature vectors alone is effective for detecting anomalies, while incorporating other modalities further improves detection performance, highlighting the benefits of multimodal feature reconstruction for robust anomaly detection in human-robot collaboration.

Related papers

• Towards a Virtual Neuroscientist: Autonomous Neuroimaging Analysis via Multi-Agent Collaboration [53.772014300855375]
  We introduce NIAgent, a multi-agent system for autonomous end-to-end neuroimaging analysis.<n>Unlike conventional flat tool-calling agents, NIAgent adopts a code-centric execution paradigm.<n>We propose a hierarchical verification framework for autonomous quality control, integrating cohort-level metric screening with agentic visual inspection.
  arXiv  Detail & Related papers  (2026-05-10T06:30:19Z)
• Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models [32.17113044802459]
  We propose a vision framework for event-level detection and direction classification of instrument handovers in surgical videos.<n>A unified multi-task formulation jointly predicts handover occurrence and interaction direction.<n>Experiments on a dataset of kidney transplant procedures demonstrate strong performance.
  arXiv  Detail & Related papers  (2026-04-08T20:31:28Z)
• Pro-HOI: Perceptive Root-guided Humanoid-Object Interaction [57.252599851031505]
  We introduce Perceptive Root-guided Humanoid-Object Interaction, Pro-HOI.<n>Pro-HOI is a generalizable framework for robust humanoid loco-manipulation.<n> Empirical validation on a Unitree G1 robot demonstrates that Pro-HOI significantly outperforms baselines in generalization and robustness.
  arXiv  Detail & Related papers  (2026-03-01T14:19:59Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection [21.460727996614704]
  MIS robots have long kinematic chains and partial visibility of their degrees of freedom in the camera.<n>We propose a novel framework that unifies the detection of geometric primitives through a shared encoding.<n>This architecture detects both keypoints and edges in a single inference and is trained on large-scale synthetic data with projective labeling.
  arXiv  Detail & Related papers  (2025-10-03T22:03:28Z)
• Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection [122.4894940892536]
  We present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level. In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss.
  arXiv  Detail & Related papers  (2022-09-25T04:56:10Z)
• An Outlier Exposure Approach to Improve Visual Anomaly Detection Performance for Mobile Robots [76.36017224414523]
  We consider the problem of building visual anomaly detection systems for mobile robots. Standard anomaly detection models are trained using large datasets composed only of non-anomalous data. We tackle the problem of exploiting these data to improve the performance of a Real-NVP anomaly detection model.
  arXiv  Detail & Related papers  (2022-09-20T15:18:13Z)
• Proactive Anomaly Detection for Robot Navigation with Multi-Sensor Fusion [7.293053431456775]
  Mobile robots produce anomalous behaviors that can lead to navigation failures. Reactive anomaly detection methods identify anomalous task executions based on the current robot state. We propose a proactive anomaly detection network (PAAD) for robot navigation in unstructured and uncertain environments.
  arXiv  Detail & Related papers  (2022-04-03T19:48:40Z)
• Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction [55.569050872780224]
  We present an online framework for safe crowd-robot interaction based on risk-sensitive optimal control, wherein the risk is modeled by the entropic risk measure. Our modular approach decouples the crowd-robot interaction into learning-based prediction and model-based control. A simulation study and a real-world experiment show that the proposed framework can accomplish safe and efficient navigation while avoiding collisions with more than 50 humans in the scene.
  arXiv  Detail & Related papers  (2020-09-12T02:02:52Z)
• Wavelet-based temporal models of human activity for anomaly detection in smart robot-assisted environments [2.299866262521074]
  This paper presents a new approach for temporal modelling of long-term human activities with smart-home sensors. The model is based on wavelet transforms and used to forecast smart sensor data, providing a temporal prior to detect unexpected events in human environments.
  arXiv  Detail & Related papers  (2020-02-26T14:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.