Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment

• URL: http://arxiv.org/abs/2007.09053v1
• Date: Mon, 13 Jul 2020 16:37:01 GMT
• Title: Situated Multimodal Control of a Mobile Robot: Navigation through a Virtual Environment
• Abstract summary: We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an embodied simulation of what the robot has encountered while exploring, and a cross-platform bridge facilitating generic communication.
• Score: 7.387138888226306
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a new interface for controlling a navigation robot in novel environments using coordinated gesture and language. We use a TurtleBot3 robot with a LIDAR and a camera, an embodied simulation of what the robot has encountered while exploring, and a cross-platform bridge facilitating generic communication. A human partner can deliver instructions to the robot using spoken English and gestures relative to the simulated environment, to guide the robot through navigation tasks.

Related papers

• A Glimpse into Long-term Physical Coexistence with Intelligent Robots [12.275897522668858]
  PHILIA is a multi-robot agent built around a robot gateway abstraction.<n>It decouples low-frequency, high-semantic agent reasoning from high-frequency, low-level robot execution.<n>We present representative use cases in which agent memory and scene understanding are grounded in robot actions.
  arXiv  Detail & Related papers  (2026-07-13T10:40:32Z)
• From Woofs to Words: Towards Intelligent Robotic Guide Dogs with Verbal Communication [6.5337939793709445]
  A robotic guide dog is an assistive quadruped robot that helps visually impaired people in obstacle avoidance and navigation.<n>We develop a novel dialog system for robotic guide dogs that uses LLMs to verbalize both navigational plans and scenes.
  arXiv  Detail & Related papers  (2026-03-13T02:11:00Z)
• Robi Butler: Multimodal Remote Interaction with a Household Robot Assistant [19.28028454264679]
  Robi Butler is a new household robot assistant that enables seamless multimodal remote interaction. It allows the human user to monitor its environment from a first-person view, issue voice or text commands, and specify target objects through hand-pointing gestures.
  arXiv  Detail & Related papers  (2024-09-30T17:49:09Z)
• Unifying 3D Representation and Control of Diverse Robots with a Single Camera [48.279199537720714]
  We introduce Neural Jacobian Fields, an architecture that autonomously learns to model and control robots from vision alone. Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot.
  arXiv  Detail & Related papers  (2024-07-11T17:55:49Z)
• LLM Granularity for On-the-Fly Robot Control [3.5015824313818578]
  In circumstances where visuals become unreliable or unavailable, can we rely solely on language to control robots? This work takes the initial steps to answer this question by: 1) evaluating the responses of assistive robots to language prompts of varying granularities; and 2) exploring the necessity and feasibility of controlling the robot on-the-fly.
  arXiv  Detail & Related papers  (2024-06-20T18:17:48Z)
• ImitationNet: Unsupervised Human-to-Robot Motion Retargeting via Shared Latent Space [9.806227900768926]
  This paper introduces a novel deep-learning approach for human-to-robot motion. Our method does not require paired human-to-robot data, which facilitates its translation to new robots. Our model outperforms existing works regarding human-to-robot similarity in terms of efficiency and precision.
  arXiv  Detail & Related papers  (2023-09-11T08:55:04Z)
• Gesture2Path: Imitation Learning for Gesture-aware Navigation [54.570943577423094]
  We present Gesture2Path, a novel social navigation approach that combines image-based imitation learning with model-predictive control. We deploy our method on real robots and showcase the effectiveness of our approach for the four gestures-navigation scenarios.
  arXiv  Detail & Related papers  (2022-09-19T23:05:36Z)
• GenLoco: Generalized Locomotion Controllers for Quadrupedal Robots [87.32145104894754]
  We introduce a framework for training generalized locomotion (GenLoco) controllers for quadrupedal robots. Our framework synthesizes general-purpose locomotion controllers that can be deployed on a large variety of quadrupedal robots. We show that our models acquire more general control strategies that can be directly transferred to novel simulated and real-world robots.
  arXiv  Detail & Related papers  (2022-09-12T15:14:32Z)
• Body Gesture Recognition to Control a Social Robot [5.557794184787908]
  We propose a gesture based language to allow humans to interact with robots using their body in a natural way. We have created a new gesture detection model using neural networks and a custom dataset of humans performing a set of body gestures to train our network.
  arXiv  Detail & Related papers  (2022-06-15T13:49:22Z)
• Socially Compliant Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation [92.66286342108934]
  Social navigation is the capability of an autonomous agent, such as a robot, to navigate in a'socially compliant' manner in the presence of other intelligent agents such as humans. Our dataset contains 8.7 hours, 138 trajectories, 25 miles of socially compliant, human teleoperated driving demonstrations.
  arXiv  Detail & Related papers  (2022-03-28T19:09:11Z)
• The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation [35.0500437688086]
  We present a benchmark of 169 natural language dialogs between a human Driver controlling a robot and a human Commander providing guidance towards navigation goals. In each trial, the pair first cooperates to localize the robot on a global map visible to the Commander, then the Driver follows Commander instructions to move the robot to a sequence of target objects. We present an initial model for the NDH task, and show that an agent trained in simulation can follow the RobotSlang dialog-based navigation instructions for controlling a physical robot platform.
  arXiv  Detail & Related papers  (2020-10-23T19:58:17Z)
• OpenBot: Turning Smartphones into Robots [95.94432031144716]
  Current robots are either expensive or make significant compromises on sensory richness, computational power, and communication capabilities. We propose to leverage smartphones to equip robots with extensive sensor suites, powerful computational abilities, state-of-the-art communication channels, and access to a thriving software ecosystem. We design a small electric vehicle that costs $50 and serves as a robot body for standard Android smartphones.
  arXiv  Detail & Related papers  (2020-08-24T18:04:50Z)
• Morphology-Agnostic Visual Robotic Control [76.44045983428701]
  MAVRIC is an approach that works with minimal prior knowledge of the robot's morphology. We demonstrate our method on visually-guided 3D point reaching, trajectory following, and robot-to-robot imitation.
  arXiv  Detail & Related papers  (2019-12-31T15:45:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.