"I'm Not Mad, Just Focused'': Understanding Human Emotions in Human-Robot Collaboration

• URL: http://arxiv.org/abs/2605.16816v1
• Date: Sat, 16 May 2026 05:18:22 GMT
• Title: "I'm Not Mad, Just Focused'': Understanding Human Emotions in Human-Robot Collaboration
• Abstract summary: We propose a novel vision language model (VLM)-based emotion recognition system for human-robot collaboration (HRC)<n>We first evaluate the VLM-ER system by assessing its semantic and sentiment similarity with human annotations on an existing HRC dataset.<n>In a user study with a service robot in a collaborative delivery task, we evaluate the effects of modulating the robot's behaviour based on the user's emotional state inferred by the VLM-ER system.
• Score: 0.5522626742911786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-robot collaboration (HRC) can benefit from robots' abilities to interpret human emotional states. However, current emotion recognition (ER) models in HRC often fall short, particularly due to their reliance on acted datasets and single-modality inputs like facial expressions. We propose a novel vision language model (VLM)-based ER system that leverages contextual understanding to improve emotion interpretation in HRC. We first evaluate the VLM-ER system by assessing its semantic and sentiment similarity with human annotations on an existing HRC dataset. Then, in a user study with a service robot in a collaborative delivery task, we evaluate the effects of modulating the robot's behaviour based on the user's emotional state inferred by the VLM-ER system. The results show that the proposed VLM-ER system achieves higher semantic similarity and positive sentiment alignment with human annotations compared to a baseline convolutional neural network-based system. Further, participants in the user study preferred emotion-adaptive robot behaviour facilitated by the VLM-ER system.

Related papers

• Robotic Grasping and Placement Controlled by EEG-Based Hybrid Visual and Motor Imagery [64.82869118243723]
  We present a framework that integrates EEG-based visual and motor imagery (VI/MI) with robotic control to enable real-time, intention-driven grasping and placement.<n>Motivated by the promise of BCI-driven robotics, this system bridges neural signals with physical control by deploying offline-pretrained decoders in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-03-03T17:41:42Z)
• Theory of Mind for Explainable Human-Robot Interaction [11.35521581811494]
  Theory of Mind (ToM) is intended to serve as a user-friendly backend to the interface of robotic systems.<n>XAI aims to make AI systems transparent and interpretable, allowing humans to understand and interact with them effectively.
  arXiv  Detail & Related papers  (2025-12-29T14:09:05Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• FABG : End-to-end Imitation Learning for Embodied Affective Human-Robot Interaction [3.8177867835232004]
  This paper proposes FABG (Facial Affective Behavior Generation), an end-to-end imitation learning system for human-robot interaction.<n>We develop an immersive virtual reality (VR) demonstration system that allows operators to perceive stereoscopic environments.<n>We deploy FABG on a real-world 25-degree-of-freedom humanoid robot, validating its effectiveness through four fundamental interaction tasks.
  arXiv  Detail & Related papers  (2025-03-03T09:58:04Z)
• HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teams [2.6627293764668902]
  This paper describes a cognitive-robotic architecture that integrates the OntoAgent cognitive framework with general-purpose robot control systems applied to human-robot teaming (HRT)<n>HarMONIC incorporates metacognition, meaningful natural language communication, and explainability capabilities required for developing mutual trust in HRT.
  arXiv  Detail & Related papers  (2024-09-26T16:48:21Z)
• Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task [17.190635800969456]
  In this paper, we examine using Large Language Models to infer human intention in a collaborative object categorization task with a physical robot.<n>We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions.
  arXiv  Detail & Related papers  (2024-04-12T12:15:14Z)
• Data-driven emotional body language generation for social robotics [58.88028813371423]
  In social robotics, endowing humanoid robots with the ability to generate bodily expressions of affect can improve human-robot interaction and collaboration. We implement a deep learning data-driven framework that learns from a few hand-designed robotic bodily expressions. The evaluation study found that the anthropomorphism and animacy of the generated expressions are not perceived differently from the hand-designed ones.
  arXiv  Detail & Related papers  (2022-05-02T09:21:39Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• A MultiModal Social Robot Toward Personalized Emotion Interaction [1.2183405753834562]
  This study demonstrates a multimodal human-robot interaction (HRI) framework with reinforcement learning to enhance the robotic interaction policy. The goal is to apply this framework in social scenarios that can let the robots generate a more natural and engaging HRI framework.
  arXiv  Detail & Related papers  (2021-10-08T00:35:44Z)
• Cognitive architecture aided by working-memory for self-supervised multi-modal humans recognition [54.749127627191655]
  The ability to recognize human partners is an important social skill to build personalized and long-term human-robot interactions. Deep learning networks have achieved state-of-the-art results and demonstrated to be suitable tools to address such a task. One solution is to make robots learn from their first-hand sensory data with self-supervision.
  arXiv  Detail & Related papers  (2021-03-16T13:50:24Z)
• Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks [83.37025218216888]
  We propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations. The robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications. Results show that the generated explanations of our approach significantly improves the collaboration performance and user perception of the robot.
  arXiv  Detail & Related papers  (2020-07-24T23:35:03Z)
• Human Perception of Intrinsically Motivated Autonomy in Human-Robot Interaction [2.485182034310304]
  A challenge in using robots in human-inhabited environments is to design behavior that is engaging, yet robust to the perturbations induced by human interaction. Our idea is to imbue the robot with intrinsic motivation (IM) so that it can handle new situations and appears as a genuine social other to humans. This article presents a "robotologist" study design that allows comparing autonomously generated behaviors with each other.
  arXiv  Detail & Related papers  (2020-02-14T09:49:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.