DeComplex: Task planning from complex natural instructions by a collocating robot

• URL: http://arxiv.org/abs/2008.10084v1
• Date: Sun, 23 Aug 2020 18:10:24 GMT
• Title: DeComplex: Task planning from complex natural instructions by a collocating robot
• Authors: Pradip Pramanick, Hrishav Bakul Barua, Chayan Sarkar
• Abstract summary: It is not trivial to execute the human intended tasks as natural language expressions can have large linguistic variations. Existing works assume either single task instruction is given to the robot at a time or there are multiple independent tasks in an instruction. We propose a method to find the intended order of execution of multiple inter-dependent tasks given in natural language instruction.
• Score: 3.158346511479111
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As the number of robots in our daily surroundings like home, office, restaurants, factory floors, etc. are increasing rapidly, the development of natural human-robot interaction mechanism becomes more vital as it dictates the usability and acceptability of the robots. One of the valued features of such a cohabitant robot is that it performs tasks that are instructed in natural language. However, it is not trivial to execute the human intended tasks as natural language expressions can have large linguistic variations. Existing works assume either single task instruction is given to the robot at a time or there are multiple independent tasks in an instruction. However, complex task instructions composed of multiple inter-dependent tasks are not handled efficiently in the literature. There can be ordering dependency among the tasks, i.e., the tasks have to be executed in a certain order or there can be execution dependency, i.e., input parameter or execution of a task depends on the outcome of another task. Understanding such dependencies in a complex instruction is not trivial if an unconstrained natural language is allowed. In this work, we propose a method to find the intended order of execution of multiple inter-dependent tasks given in natural language instruction. Based on our experiment, we show that our system is very accurate in generating a viable execution plan from a complex instruction.

Related papers

• COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models [49.24666980374751]
  COHERENT is a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems. A Proposal-Execution-Feedback-Adjustment mechanism is designed to decompose and assign actions for individual robots. The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency.
  arXiv  Detail & Related papers  (2024-09-23T15:53:41Z)
• LEMMA: Learning Language-Conditioned Multi-Robot Manipulation [21.75163634731677]
  LanguagE-Conditioned Multi-robot MAnipulation (LEMMA) LeMMA features 8 types of procedurally generated tasks with varying degree of complexity. For each task, we provide 800 expert demonstrations and human instructions for training and evaluations.
  arXiv  Detail & Related papers  (2023-08-02T04:37:07Z)
• Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks [21.65346551790888]
  DeL-TaCo is a method for conditioning a robotic policy on task embeddings comprised of two components: a visual demonstration and a language instruction. To our knowledge, this is the first work to show that simultaneously conditioning a multi-task robotic manipulation policy on both demonstration and language embeddings improves sample efficiency and generalization over conditioning on either modality alone.
  arXiv  Detail & Related papers  (2022-10-10T08:06:58Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Interactive Hierarchical Guidance using Language [5.797847756967884]
  We introduce an approach where we use language to specify sub-tasks and a high-level planner issues language commands to a low level controller. Our experiments show that this method is able to solve complex long horizon planning tasks with limited human supervision.
  arXiv  Detail & Related papers  (2021-10-09T21:34:32Z)
• Lifelong Robotic Reinforcement Learning by Retaining Experiences [61.79346922421323]
  Many multi-task reinforcement learning efforts assume the robot can collect data from all tasks at all times. In this work, we study a practical sequential multi-task RL problem motivated by the practical constraints of physical robotic systems. We derive an approach that effectively leverages the data and policies learned for previous tasks to cumulatively grow the robot's skill-set.
  arXiv  Detail & Related papers  (2021-09-19T18:00:51Z)
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation [80.29069988090912]
  We study the problem of learning a range of vision-based manipulation tasks from a large offline dataset of robot interaction. We propose to leverage offline robot datasets with crowd-sourced natural language labels. We find that our approach outperforms both goal-image specifications and language conditioned imitation techniques by more than 25%.
  arXiv  Detail & Related papers  (2021-09-02T17:42:13Z)
• A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution [54.385344986265714]
  We propose a persistent spatial semantic representation method to bridge the gap between language and robot actions. We evaluate our approach on the ALFRED benchmark and achieve state-of-the-art results, despite completely avoiding the commonly used step-by-step instructions.
  arXiv  Detail & Related papers  (2021-07-12T17:47:19Z)
• Enabling human-like task identification from natural conversation [7.00597813134145]
  We provide a non-trivial method to combine an NLP engine and a planner such that a robot can successfully identify tasks and all the relevant parameters and generate an accurate plan for the task. This work makes a significant stride towards enabling a human-like task understanding capability in a robot.
  arXiv  Detail & Related papers  (2020-08-23T17:19:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.