Hierarchical Policies from Verbal and Egocentric Human Signals for Natural Human-Robot Interaction

• URL: http://arxiv.org/abs/2606.10276v1
• Date: Tue, 09 Jun 2026 00:50:29 GMT
• Title: Hierarchical Policies from Verbal and Egocentric Human Signals for Natural Human-Robot Interaction
• Abstract summary: We present EDITH, a robot framework that captures the human's nonverbal signals through continuous streams of first-person view and gaze from smart glasses.<n>Our hardware system streams the human's first-person view, gaze, and speech to the robot in real time, transcribing the speech into language instructions.<n>In experiments on human-robot interactive tasks, EDITH enables the robot to act on the human's nonverbal signals even when intent is expressed only briefly.
• Score: 28.306877525705236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For natural human-robot interaction, a robot must understand human intent expressed not only through language but also through nonverbal signals such as gestures and gaze. However, current robot policies rely on language instructions as the sole interface for conveying intent, leaving nonverbal signals unused and placing the full burden of communication. In this work, we present EDITH, a robot framework that captures the human's nonverbal signals through continuous streams of first-person view and gaze from smart glasses, and uses them alongside language instructions as inputs to the robot policy. Our hardware system streams the human's first-person view, gaze, and speech to the robot in real time, transcribing the speech into language instructions. To handle these rich but noisy signals, we design a hierarchical policy in which a high-level policy infers the human's intent and produces a sequence of subtasks, where each subtask is represented as a fine-grained instruction paired with a keyframe that grounds the intent in the scene (e.g., the frame where the human points at the target object). A low-level policy then executes these subtasks. In our experiments on human-robot interactive tasks, EDITH enables the robot to act on the human's nonverbal signals even when intent is expressed only briefly, and significantly reduces user effort to convey intent compared to using language instructions alone. Visit our project page for source code and real-robot demo videos.

Related papers

• LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music [1.7188280334580195]
  This paper introduces a novel framework that synthesizes complex robotic actions from a rich tapestry of multimodal human inputs.<n>Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection.<n>The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music.
  arXiv  Detail & Related papers  (2026-06-30T05:35:26Z)
• Distill: Uncovering the True Intent behind Human-Robot Communication [3.8732212038055827]
  We propose the Distill approach for human-robot communication interfaces.<n>Given a task specification, Distill removes unnecessary steps; generalizes the meaning behind individual steps; and relaxes ordering constraints between steps.<n>We implemented Distill on a web interface and, through a crowdsourcing study, demonstrated its ability to elicit and refine user intent from initial task specifications.
  arXiv  Detail & Related papers  (2026-05-14T02:05:49Z)
• HandMeThat: Human-Robot Communication in Physical and Social Environments [73.91355172754717]
  HandMeThat is a benchmark for a holistic evaluation of instruction understanding and following in physical and social environments. HandMeThat contains 10,000 episodes of human-robot interactions. We show that both offline and online reinforcement learning algorithms perform poorly on HandMeThat.
  arXiv  Detail & Related papers  (2023-10-05T16:14:46Z)
• HuBo-VLM: Unified Vision-Language Model designed for HUman roBOt interaction tasks [5.057755436092344]
  Human robot interaction is an exciting task, which aimed to guide robots following instructions from human. HuBo-VLM is proposed to tackle perception tasks associated with human robot interaction.
  arXiv  Detail & Related papers  (2023-08-24T03:47:27Z)
• Open-World Object Manipulation using Pre-trained Vision-Language Models [72.87306011500084]
  For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary. We develop a simple approach, which leverages a pre-trained vision-language model to extract object-identifying information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments.
  arXiv  Detail & Related papers  (2023-03-02T01:55:10Z)
• Robots with Different Embodiments Can Express and Influence Carefulness in Object Manipulation [104.5440430194206]
  This work investigates the perception of object manipulations performed with a communicative intent by two robots. We designed the robots' movements to communicate carefulness or not during the transportation of objects.
  arXiv  Detail & Related papers  (2022-08-03T13:26:52Z)
• Understanding Natural Language in Context [13.112390442564442]
  We focus on cognitive robots, which have some knowledge-based models of the world and operate by reasoning and planning with this model. Our goal in this research is to translate natural language utterances into this robot's formalism. We do so by combining off-the-shelf SOTA language models, planning tools, and the robot's knowledge-base for better communication.
  arXiv  Detail & Related papers  (2022-05-25T11:52:16Z)
• Synthesis and Execution of Communicative Robotic Movements with Generative Adversarial Networks [59.098560311521034]
  We focus on how to transfer on two different robotic platforms the same kinematics modulation that humans adopt when manipulating delicate objects. We choose to modulate the velocity profile adopted by the robots' end-effector, inspired by what humans do when transporting objects with different characteristics. We exploit a novel Generative Adversarial Network architecture, trained with human kinematics examples, to generalize over them and generate new and meaningful velocity profiles.
  arXiv  Detail & Related papers  (2022-03-29T15:03:05Z)
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation [80.29069988090912]
  We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. We propose to leverage offline robot datasets with crowd-sourced natural language labels. We find that our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%.
  arXiv  Detail & Related papers  (2021-09-02T17:42:13Z)
• Self-supervised reinforcement learning for speaker localisation with the iCub humanoid robot [58.2026611111328]
  Looking at a person's face is one of the mechanisms that humans rely on when it comes to filtering speech in noisy environments. Having a robot that can look toward a speaker could benefit ASR performance in challenging environments. We propose a self-supervised reinforcement learning-based framework inspired by the early development of humans.
  arXiv  Detail & Related papers  (2020-11-12T18:02:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.