The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models

• URL: http://arxiv.org/abs/2607.16318v1
• Date: Wed, 15 Jul 2026 06:55:11 GMT
• Title: The World According to a Social Robot -- Augmenting Human-Robot Dialogue With Vision Language Models
• Abstract summary: Vision Language Models (VLMs) enable robots to visually perceive their environment as well as the actions and characteristics of their conversation partner or humans in collaboration.<n>This paper presents initial experiences with the application of a Mistral AI language model with a Pepper robot for Human-Robot Interaction (HRI) in dialogue.<n>The results show that incorporating visual information adds context to the dialogue with only a moderate increase in response time.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision Language Models (VLMs) enable robots to visually perceive their environment as well as the actions and characteristics of their conversation partner or humans in collaboration. Especially for social robots deployed in everyday settings and for uncomplicated, natural use, it is essential that the robot has an understanding of situations that is appropriate to human customs. This paper presents initial experiences with the application of a Mistral AI language model with a Pepper robot for Human-Robot Interaction (HRI) in dialogue, as well as an investigation of the effects of additional visual information on response time in different models. The results show that incorporating visual information adds context to the dialogue with only a moderate increase in response time, enabling both the robot and the human to take into account unspoken elements of the situation. Furthermore, using an LLM hosted in Europe offers a solution that complies with European data protection regulations and can therefore facilitate real-life applications more easily.

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction [0.0]
  This work presents a novel HRI framework that combines advanced vision-language models, speech processing, and fuzzy logic.<n>The proposed system integrates Florence-2 for object detection, Llama 3.1 for natural language understanding, and Whisper for speech recognition.<n> Experimental evaluations conducted on consumer-grade hardware demonstrate a command execution accuracy of 75%.
  arXiv  Detail & Related papers  (2026-02-23T09:05:15Z)
• Using Vision-Language Models as Proxies for Social Intelligence in Human-Robot Interaction [16.018759023223502]
  We analyze how people signal interaction readiness through nonverbal behaviors and how expert wizards use these cues to guide engagement.<n>We propose a two-stage pipeline in which lightweight perceptual detectors are used to selectively trigger heavier video-based vision-language model (VLM) queries.<n>Our findings suggest that selectively using VLMs as proxies for social reasoning enables socially responsive robot behavior.
  arXiv  Detail & Related papers  (2025-12-08T05:17:58Z)
• RoboOmni: Proactive Robot Manipulation in Omni-modal Context [165.09049429566238]
  We introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.<n>We present RoboOmni, a framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution.<n>Experiments in simulation and real-world settings show that Robo Omni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
  arXiv  Detail & Related papers  (2025-10-27T18:49:03Z)
• TalkWithMachines: Enhancing Human-Robot Interaction for Interpretable Industrial Robotics Through Large/Vision Language Models [1.534667887016089]
  The presented paper investigates recent advancements in Large Language Models (LLMs) and Vision Language Models (VLMs)<n>This integration allows robots to understand and execute commands given in natural language and to perceive their environment through visual and/or descriptive inputs.<n>Our paper outlines four LLM-assisted simulated robotic control, which explore (i) low-level control, (ii) the generation of language-based feedback that describes the robot's internal states, (iii) the use of visual information as additional input, and (iv) the use of robot structure information for generating task plans and feedback.
  arXiv  Detail & Related papers  (2024-12-19T23:43:40Z)
• Project Report: Requirements for a Social Robot as an Information Provider in the Public Sector [0.0]
  We have devised an application scenario for integrating a humanoid social robot into an official environment.<n>We developed a corresponding robot application and carried out initial tests and evaluations in a project together with the Kiel City Council.<n>One of the most important insights gained in the project was that a humanoid robot with natural language processing capabilities proved to be much more preferred by users.<n>We propose a connection of the ACT-R cognitive architecture with the robot, where an ACT-R model is used in interaction with the robot application to cognitively process and enhance a dialogue between human and robot.
  arXiv  Detail & Related papers  (2024-12-06T13:07:06Z)
• Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models [81.55156507635286]
  Legged robots are physically capable of navigating a diverse variety of environments and overcoming a wide range of obstructions. Current learning methods often struggle with generalization to the long tail of unexpected situations without heavy human supervision. We propose a system, VLM-Predictive Control (VLM-PC), combining two key components that we find to be crucial for eliciting on-the-fly, adaptive behavior selection.
  arXiv  Detail & Related papers  (2024-07-02T21:00:30Z)
• Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task [17.190635800969456]
  In this paper, we examine using Large Language Models to infer human intention in a collaborative object categorization task with a physical robot.<n>We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions.
  arXiv  Detail & Related papers  (2024-04-12T12:15:14Z)
• A Sign Language Recognition System with Pepper, Lightweight-Transformer, and LLM [0.9775599530257609]
  This research explores using lightweight deep neural network architectures to enable the humanoid robot Pepper to understand American Sign Language (ASL) We introduce a lightweight and efficient model for ASL understanding optimized for embedded systems, ensuring rapid sign recognition while conserving computational resources. We tailor interactions to allow the Pepper Robot to generate natural Co-Speech Gesture responses, laying the foundation for more organic and intuitive humanoid-robot dialogues.
  arXiv  Detail & Related papers  (2023-09-28T23:54:41Z)
• WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Model [92.90127398282209]
  This paper investigates the potential of integrating the most recent Large Language Models (LLMs) and existing visual grounding and robotic grasping system. We introduce the WALL-E (Embodied Robotic WAiter load lifting with Large Language model) as an example of this integration. We deploy this LLM-empowered system on the physical robot to provide a more user-friendly interface for the instruction-guided grasping task.
  arXiv  Detail & Related papers  (2023-08-30T11:35:21Z)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control [140.48218261864153]
  We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control. Our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training.
  arXiv  Detail & Related papers  (2023-07-28T21:18:02Z)
• Open-World Object Manipulation using Pre-trained Vision-Language Models [72.87306011500084]
  For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary. We develop a simple approach, which leverages a pre-trained vision-language model to extract object-identifying information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments.
  arXiv  Detail & Related papers  (2023-03-02T01:55:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.