An Intelligent-Cloud Edge Multimodal Interaction System for Robots

• URL: http://arxiv.org/abs/2607.14675v3
• Date: Thu, 23 Jul 2026 03:08:15 GMT
• Title: An Intelligent-Cloud Edge Multimodal Interaction System for Robots
• Abstract summary: This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents.<n>The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback.
• Score: 4.1212219465063304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robust human-robot interaction in complex environments requires accurate gesture perception, semantic scene understanding, and reliable task planning under limited onboard computing resources. This paper presents a cloud-edge multimodal interaction framework that integrates an enhanced YOLO-based gesture detector with coordinated large language model (LLM) and vision-language model (VLM) agents. The proposed detector, incorporates the Convolutional Block Attention Module (CBAM) into the neck and replaces the baseline bounding-box regression objective with Distance-IoU (DIoU) loss. These modifications improve feature discrimination and localization for small or partially occluded gestures in complex backgrounds. The cloud layer performs gesture detection, scene understanding, multimodal fusion, and action planning, whereas the TonyPi robot locally handles data acquisition, communication, action execution, and feedback. Experiments on a public gesture dataset and a custom dataset show that YOLO-DC achieves precision values of 98.9% and 95.0%, with mAP@0.5 values of 90.7% and 92.7%, respectively. System-level evaluation yields success rates of 95%, 88%, and 82% for single-action, composite-action, and vision-dependent tasks. A 30 participant evaluation yields an overall mean satisfaction score of 3.69 out of 5. These results demonstrate the feasibility of combining refined gesture detection with multimodal agents for resource-constrained robotic interaction.

Related papers

• Modality-Decoupled Federated Learning for Privacy-Preserving Embodied Intelligence in 6G [90.42669277637026]
  Sixth-generation (6G) wireless networks are expected to provide a key infrastructure for large-scale embodied intelligence.<n>This article proposes FedMVLA, a modality-decoupled FL framework for privacy-preserving embodied intelligence in 6G networks.
  arXiv  Detail & Related papers  (2026-09-09T01:36:44Z)
• SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks [73.92333717662558]
  SpatialWorld is a benchmark for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks.<n>It features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration)<n>For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier.
  arXiv  Detail & Related papers  (2026-06-08T15:51:51Z)
• Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments [96.23886784364997]
  Embodied intelligence is often studied through specialized models for individual tasks such as manipulation or navigation.<n>We present Qwen-VLA, a unified embodied foundation model that extends Qwen's vision-language modeling stack to continuous action and trajectory generation.
  arXiv  Detail & Related papers  (2026-05-28T17:36:31Z)
• A Gesture-Based Visual Learning Model for Acoustophoretic Interactions using a Swarm of AcoustoBots [8.171680721155761]
  AcoustoBots are mobile acoustophoretic robots capable of delivering haptics, directional audio, and acoustic levitation.<n>Existing implementations rely on scripted commands and lack an intuitive interface for real-time human control.<n>This work presents a gesture-based visual learning framework for contactless human-swarm interaction with a multimodal AcoustoBot platform.
  arXiv  Detail & Related papers  (2026-04-21T16:32:38Z)
• Design and Validation of a Lightweight 1D CNN for Affective Touch Classification in Soft Plush Companions [59.49967357689445]
  Soft, sensorized companions offer a physically safe and emotionally intuitive interface for socially assistive technologies.<n>This study presents a complete open-source dilated real-based framework for the development and validation of deep learning models for affective touch recognition in soft interactive companions.
  arXiv  Detail & Related papers  (2026-04-16T10:20:05Z)
• Active Stereo-Camera Outperforms Multi-Sensor Setup in ACT Imitation Learning for Humanoid Manipulation [4.864819846886142]
  This paper benchmarks 14 sensor combinations on the Unitree G1 humanoid robot equipped with three-finger hands for two manipulation tasks.<n>We evaluate the integration of tactile and proprioceptive modalities alongside active vision.
  arXiv  Detail & Related papers  (2026-03-30T13:30:34Z)
• Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition [71.5328300638085]
  Zero-shot Human-object interaction (HOI) detection aims to locate humans and objects in images and recognize their interactions.<n>Existing methods, including two-stage methods, tightly couple interaction recognition with a specific detector.<n>We propose a decoupled framework that separates object detection from IR and leverages multi-modal large language models (MLLMs) for zero-shot IR.
  arXiv  Detail & Related papers  (2026-02-16T19:01:31Z)
• Real-Time Human-Robot Interaction Intent Detection Using RGB-based Pose and Emotion Cues with Cross-Camera Model Generalization [0.8839687029212673]
  Service robots in public spaces require real-time understanding of human behavioral intentions for natural interaction.<n>We present a framework for frame-accurate human-robot interaction intent detection that fuses camera-invariant 2D skeletal pose and facial emotion features extracted from monocular RGB video.
  arXiv  Detail & Related papers  (2025-12-18T08:44:22Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• YOLO-APD: Enhancing YOLOv8 for Robust Pedestrian Detection on Complex Road Geometries [0.0]
  This paper introduces YOLO-APD, a novel deep learning architecture enhancing the YOLOv8 framework specifically for this challenge.<n>YOLO-APD achieves state-of-the-art detection accuracy, reaching 77.7% mAP@0.5:0.95 and exceptional pedestrian recall exceeding 96%.<n>It maintains real-time processing capabilities at 100 FPS, showcasing a superior balance between accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-07-07T18:03:40Z)
• λ: A Benchmark for Data-Efficiency in Long-Horizon Indoor Mobile Manipulation Robotics [11.901933884058021]
  We introduce the LAMBDA benchmark-Long-horizon Actions for Mobile-manipulation Benchmarking of Directed Activities.<n>Our benchmark includes 571 human-collected demonstrations that provide realism and diversity in simulated and real-world settings.<n>We find that learning methods, even when pretrained, yield lower success rates, while a neuro-symbolic method performs significantly better and requires less data.
  arXiv  Detail & Related papers  (2024-11-28T19:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.