Navigating to Success in Multi-Modal Human-Robot Collaboration: Analysis and Corpus Release

• URL: http://arxiv.org/abs/2310.17568v1
• Date: Thu, 26 Oct 2023 16:56:01 GMT
• Title: Navigating to Success in Multi-Modal Human-Robot Collaboration: Analysis and Corpus Release
• Abstract summary: Human-guided robotic exploration is a useful approach to gathering information at remote locations. Maintaining common ground between the remotely-located partners is a challenge. We explore how participants utilized multiple modalities to investigate a remote location with the help of a robotic partner.
• Score: 2.467008547326945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-guided robotic exploration is a useful approach to gathering information at remote locations, especially those that might be too risky, inhospitable, or inaccessible for humans. Maintaining common ground between the remotely-located partners is a challenge, one that can be facilitated by multi-modal communication. In this paper, we explore how participants utilized multiple modalities to investigate a remote location with the help of a robotic partner. Participants issued spoken natural language instructions and received from the robot: text-based feedback, continuous 2D LIDAR mapping, and upon-request static photographs. We noticed that different strategies were adopted in terms of use of the modalities, and hypothesize that these differences may be correlated with success at several exploration sub-tasks. We found that requesting photos may have improved the identification and counting of some key entities (doorways in particular) and that this strategy did not hinder the amount of overall area exploration. Future work with larger samples may reveal the effects of more nuanced photo and dialogue strategies, which can inform the training of robotic agents. Additionally, we announce the release of our unique multi-modal corpus of human-robot communication in an exploration context: SCOUT, the Situated Corpus on Understanding Transactions.

Related papers

• Communication-Constrained Multi-Robot Exploration With Adaptive Communication Windows [16.619647268424412]
  This paper introduces MACE, a decentralized exploration framework that actively evaluates whether establishing communication is worthwhile.<n>We demonstrate that MACE reduces the total exploration time by up to 23% compared to existing communication-constrained exploration strategies.
  arXiv  Detail & Related papers  (2026-09-11T07:05:42Z)
• Robots Ask the Way: Communication-Enabled Social Navigation [12.914876079465431]
  We introduce Communication-enabled Social Navigation (CommNav)<n>In this novel task, robotic agents actively seek assistance from residents to locate target individuals.<n>We show that explicit human-robot communication substantially enhances multi-person navigation performance.
  arXiv  Detail & Related papers  (2026-07-01T15:07:40Z)
• Enhancing Explainability with Multimodal Context Representations for Smarter Robots [0.0]
  Key issue in Human-Robot Interaction is enabling robots to effectively perceive and reason over multimodal inputs, such as audio and vision. We propose a generalized and explainable multimodal framework for context representation, designed to improve the fusion of speech and vision modalities.
  arXiv  Detail & Related papers  (2025-02-28T13:36:47Z)
• NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction [24.649184273236237]
  This dataset serves as a foundational resource for research at the intersection of HRI and machine learning. By providing multimodal inputs and detailed annotations, NatSGLD enables exploration in areas such as multimodal instruction following, plan recognition, and human-advisable reinforcement learning from demonstrations.
  arXiv  Detail & Related papers  (2025-02-23T21:27:06Z)
• CuriousBot: Interactive Mobile Exploration via Actionable 3D Relational Object Graph [12.54884302440877]
  Mobile exploration is a longstanding challenge in robotics. Existing robotic exploration approaches via active interaction are often restricted to tabletop scenes. We introduce a 3D relational object graph that encodes diverse object relations and enables exploration through active interaction.
  arXiv  Detail & Related papers  (2025-01-23T02:39:04Z)
• Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Grounding [85.63710017456792]
  FuSe is a novel approach that enables finetuning visuomotor generalist policies on heterogeneous sensor modalities. We show that FuSe enables performing challenging tasks that require reasoning jointly over modalities such as vision, touch, and sound. Experiments in the real world show that FuSeis able to increase success rates by over 20% compared to all considered baselines.
  arXiv  Detail & Related papers  (2025-01-08T18:57:33Z)
• Human-Robot Dialogue Annotation for Multi-Modal Common Ground [4.665414514091581]
  We describe the development of symbolic representations annotated on human-robot dialogue data to make dimensions of meaning accessible to autonomous systems participating in collaborative, natural language dialogue, and to enable common ground with human partners. A particular challenge for establishing common ground arises in remote dialogue, where a human and robot are engaged in a joint navigation and exploration task of an unfamiliar environment, but where the robot cannot immediately share high quality visual information due to limited communication constraints. Within this paradigm, we capture propositional semantics and the illocutionary force of a single utterance within the dialogue through our Dialogue-AMR annotation, an augmentation of Abstract Meaning Representation
  arXiv  Detail & Related papers  (2024-11-19T19:33:54Z)
• A Multi-Modal Explainability Approach for Human-Aware Robots in Multi-Party Conversation [39.87346821309096]
  We present an addressee estimation model with improved performance in comparison with the previous SOTA. We also propose several ways to incorporate explainability and transparency in the aforementioned architecture.
  arXiv  Detail & Related papers  (2024-05-20T13:09:32Z)
• Self-Explainable Affordance Learning with Embodied Caption [63.88435741872204]
  We introduce Self-Explainable Affordance learning (SEA) with embodied caption. SEA enables robots to articulate their intentions and bridge the gap between explainable vision-language caption and visual affordance learning. We propose a novel model to effectively combine affordance grounding with self-explanation in a simple but efficient manner.
  arXiv  Detail & Related papers  (2024-04-08T15:22:38Z)
• NatSGD: A Dataset with Speech, Gestures, and Demonstrations for Robot Learning in Natural Human-Robot Interaction [19.65778558341053]
  Speech-gesture HRI datasets often focus on elementary tasks, like object pointing and pushing. We introduce NatSGD, a multimodal HRI dataset encompassing human commands through speech and gestures. We demonstrate its effectiveness in training robots to understand tasks through multimodal human commands.
  arXiv  Detail & Related papers  (2024-03-04T18:02:41Z)
• iCub! Do you recognize what I am doing?: multimodal human action recognition on multisensory-enabled iCub robot [0.0]
  We show that the proposed multimodal ensemble learning leverages complementary characteristics of three color cameras and one depth sensor. The results indicate that the proposed models can be deployed on the iCub robot that requires multimodal action recognition.
  arXiv  Detail & Related papers  (2022-12-17T12:40:54Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Gesture2Path: Imitation Learning for Gesture-aware Navigation [54.570943577423094]
  We present Gesture2Path, a novel social navigation approach that combines image-based imitation learning with model-predictive control. We deploy our method on real robots and showcase the effectiveness of our approach for the four gestures-navigation scenarios.
  arXiv  Detail & Related papers  (2022-09-19T23:05:36Z)
• Incremental 3D Scene Completion for Safe and Efficient Exploration Mapping and Planning [60.599223456298915]
  We propose a novel way to integrate deep learning into exploration by leveraging 3D scene completion for informed, safe, and interpretable mapping and planning. We show that our method can speed up coverage of an environment by 73% compared to the baselines with only minimal reduction in map accuracy. Even if scene completions are not included in the final map, we show that they can be used to guide the robot to choose more informative paths, speeding up the measurement of the scene with the robot's sensors by 35%.
  arXiv  Detail & Related papers  (2022-08-17T14:19:33Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• Open-Ended Multi-Modal Relational Reasoning for Video Question Answering [1.8699569122464073]
  The primary focus of this agent is to assist individuals using language-based interactions within video-based scenes. Our proposed method integrates video recognition technology and natural language processing models within the robotic agent.
  arXiv  Detail & Related papers  (2020-12-01T20:49:59Z)
• Batch Exploration with Examples for Scalable Robotic Reinforcement Learning [63.552788688544254]
  Batch Exploration with Examples (BEE) explores relevant regions of the state-space guided by a modest number of human provided images of important states. BEE is able to tackle challenging vision-based manipulation tasks both in simulation and on a real Franka robot.
  arXiv  Detail & Related papers  (2020-10-22T17:49:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.