From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping

• URL: http://arxiv.org/abs/2607.00530v1
• Date: Wed, 01 Jul 2026 07:19:00 GMT
• Title: From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping
• Abstract summary: This paper investigates whether a 15 percentage point gain in end-to-end task success is sufficient to produce consistent and measurable differences in user perception.<n>The baseline system combines Whisper for speech recognition, Florence-2 for open-vocabulary object detection, LLaMA 3.1 for action extraction, and an interval Type-2 fuzzy logic controller for motion execution.
• Score: 11.17111944571978
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Improvements in the technical performance of human--robot interaction (HRI) systems do not automatically translate into differences that human users can detect during live interaction. This paper investigates whether a 15 percentage point gain in end-to-end task success (from 75% in a multimodal baseline system to 90% in an improved configuration identified through a prior ablation study) is sufficient to produce consistent and measurable differences in user perception. The baseline system combines Whisper for speech recognition, Florence-2 for open-vocabulary object detection, LLaMA 3.1 for action extraction, and an interval Type-2 fuzzy logic controller for motion execution. The improved configuration replaces the perception and language modules with Grounding DINO + SAM and Qwen 3.5 9B, respectively, while retaining the same controller. A within-subject user study with 24 participants compared both systems on the same tabletop object-grasping task. After interacting with each configuration, participants rated perceived speed, reliability, and overall competence and fluency on a 7-point Likert scale. Results show that 17 out of 24 participants (70.83%) preferred the improved system (exact binomial test, p = 0.043, h = 0.43), and all three perceptual constructs were rated significantly higher for the improved configuration after Holm correction, with large to very large effect sizes (p < 0.001). These findings confirm that the identified technical improvements are perceptible to users in direct interaction and underscore the importance of complementing benchmark evaluation with user-centred evidence when assessing robotic manipulation pipelines.

Related papers

• An Intelligent-Cloud Edge Multimodal Interaction System for Robots [4.1212219465063304]
  This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents.<n>The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback.
  arXiv  Detail & Related papers  (2026-07-16T07:39:18Z)
• An Interpretable Closed-Loop Intelligent Tutoring System for Multimodal Affective Feedback in Asynchronous Presentation Training [0.0]
  The ITS maps multimodal inputs into evidence-based feedback that can be traced back to observable performance cues.<n>The system achieved rubric-aligned scoring with performance levels comparable to expert ratings.
  arXiv  Detail & Related papers  (2026-05-17T14:12:40Z)
• Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task [0.0]
  This manuscript extends our previous multimodal human-robot interaction system by introducing a controlled ablation study of the three modules that most strongly influence end-to-end performance.<n>The goal is not to redesign the full pipeline, but to isolate the contribution of each component under a common experimental protocol and then evaluate the best combinations end-to-end.
  arXiv  Detail & Related papers  (2026-05-01T15:04:53Z)
• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions [59.49967357689445]
  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies.<n>This study presents a complete open-source dilated real-based framework for the development and validation of deep learning models for affective touch recognition in soft interactive companions.
  arXiv  Detail & Related papers  (2026-04-16T10:20:05Z)
• Predicting the Intention to Interact with a Service Robot:the Role of Gaze Cues [51.58558750517068]
  Service robots need to perceive as early as possible that an approaching person intends to interact. We solve this perception task with a sequence-to-sequence classifier of a potential user intention to interact. Our main contribution is a study of the benefit of features representing the person's gaze in this context.
  arXiv  Detail & Related papers  (2024-04-02T14:22:54Z)
• MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition [62.89464258519723]
  We propose a multi-layer cross-attention fusion based AVSR approach that promotes representation of each modality by fusing them at different levels of audio/visual encoders. Our proposed approach surpasses the first-place system, establishing a new SOTA cpCER of 29.13% on this dataset.
  arXiv  Detail & Related papers  (2024-01-07T08:59:32Z)
• ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention [48.697458429460184]
  Two factors, information bottleneck sensitivity and inconsistency between different attention topologies, could affect the performance of the Sparse Transformer. This paper proposes a well-designed model named ERNIE-Sparse. It consists of two distinctive parts: (i) Hierarchical Sparse Transformer (HST) to sequentially unify local and global information, and (ii) Self-Attention Regularization (SAR) to minimize the distance for transformers with different attention topologies.
  arXiv  Detail & Related papers  (2022-03-23T08:47:01Z)
• Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning [69.42679922160684]
  We propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment.
  arXiv  Detail & Related papers  (2020-11-01T19:50:34Z)
• Effects of Word-frequency based Pre- and Post- Processings for Audio Captioning [49.41766997393417]
  The system we used for Task 6 (Automated Audio Captioning)of the Detection and Classification of Acoustic Scenes and Events(DCASE) 2020 Challenge combines three elements, namely, dataaugmentation, multi-task learning, and post-processing, for audiocaptioning. The system received the highest evaluation scores, but which of the individual elements most fully contributed to its perfor-mance has not yet been clarified.
  arXiv  Detail & Related papers  (2020-09-24T01:07:33Z)
• Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy [8.11839312231511]
  Mixed-initiative systems allow users to interactively provide feedback to improve system performance. Our research investigates how the act of providing feedback can affect user understanding of an intelligent system and its accuracy.
  arXiv  Detail & Related papers  (2020-08-28T16:46:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.