Multimodal Contextualized Plan Prediction for Embodied Task Completion
- URL: http://arxiv.org/abs/2305.06485v1
- Date: Wed, 10 May 2023 22:29:12 GMT
- Title: Multimodal Contextualized Plan Prediction for Embodied Task Completion
- Authors: Mert \.Inan, Aishwarya Padmakumar, Spandana Gella, Patrick Lange,
Dilek Hakkani-Tur
- Abstract summary: Task planning is an important component of traditional robotics systems enabling robots to compose fine grained skills to perform more complex tasks.
Recent work building systems for translating natural language to executable actions for task completion in simulated embodied agents is focused on directly predicting low level action sequences.
We focus on predicting a higher level plan representation for one such embodied task completion dataset - TEACh.
- Score: 9.659463406886301
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Task planning is an important component of traditional robotics systems
enabling robots to compose fine grained skills to perform more complex tasks.
Recent work building systems for translating natural language to executable
actions for task completion in simulated embodied agents is focused on directly
predicting low level action sequences that would be expected to be directly
executable by a physical robot. In this work, we instead focus on predicting a
higher level plan representation for one such embodied task completion dataset
- TEACh, under the assumption that techniques for high-level plan prediction
from natural language are expected to be more transferable to physical robot
systems. We demonstrate that better plans can be predicted using multimodal
context, and that plan prediction and plan execution modules are likely
dependent on each other and hence it may not be ideal to fully decouple them.
Further, we benchmark execution of oracle plans to quantify the scope for
improvement in plan prediction models.
Related papers
- Ask-before-Plan: Proactive Language Agents for Real-World Planning [68.08024918064503]
Proactive Agent Planning requires language agents to predict clarification needs based on user-agent conversation and agent-environment interaction.
We propose a novel multi-agent framework, Clarification-Execution-Planning (textttCEP), which consists of three agents specialized in clarification, execution, and planning.
arXiv Detail & Related papers (2024-06-18T14:07:28Z) - Safe Task Planning for Language-Instructed Multi-Robot Systems using Conformal Prediction [11.614036749291216]
We introduce a new decentralized multi-robot planner called S-ATLAS for Safe plAnning for Teams of Language-instructed AgentS.
We show that the proposed planner can achieve user-specified task success rates while minimizing the overall number of help requests.
arXiv Detail & Related papers (2024-02-23T15:02:44Z) - Consolidating Trees of Robotic Plans Generated Using Large Language
Models to Improve Reliability [6.4111574364474215]
The inherent probabilistic nature of Large Language Models (LLMs) introduces an element of unpredictability.
This paper introduces an innovative approach aims to generate correct and optimal robotic task plans for diverse real-world demands and scenarios.
arXiv Detail & Related papers (2024-01-15T18:01:59Z) - Conformal Temporal Logic Planning using Large Language Models [29.57952582715011]
This paper addresses a new motion planning problem for mobile robots tasked with accomplishing multiple high-level sub-tasks.
These sub-tasks should be accomplished in a temporal and logical order.
Our goal is to design robot plans that satisfy tasks defined over NL-based atomic propositions.
arXiv Detail & Related papers (2023-09-18T19:05:25Z) - Embodied Task Planning with Large Language Models [86.63533340293361]
We propose a TAsk Planing Agent (TaPA) in embodied tasks for grounded planning with physical scene constraint.
During inference, we discover the objects in the scene by extending open-vocabulary object detectors to multi-view RGB images collected in different achievable locations.
Experimental results show that the generated plan from our TaPA framework can achieve higher success rate than LLaVA and GPT-3.5 by a sizable margin.
arXiv Detail & Related papers (2023-07-04T17:58:25Z) - EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought [95.37585041654535]
Embodied AI is capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments.
In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI.
Experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering.
arXiv Detail & Related papers (2023-05-24T11:04:30Z) - A Framework for Neurosymbolic Robot Action Planning using Large Language Models [3.0501524254444767]
We present a framework aimed at bridging the gap between symbolic task planning and machine learning approaches.
The rationale is training Large Language Models (LLMs) into a neurosymbolic task planner compatible with the Planning Domain Definition Language (PDDL)
Preliminary results in selected domains show that our method can: (i) solve 95.5% of problems in a test data set of 1,000 samples; (ii) produce plans up to 13.5% shorter than a traditional symbolic planner; (iii) reduce average overall waiting times for a plan availability by up to 61.4%.
arXiv Detail & Related papers (2023-03-01T11:54:22Z) - ProgPrompt: Generating Situated Robot Task Plans using Large Language
Models [68.57918965060787]
Large language models (LLMs) can be used to score potential next actions during task planning.
We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
arXiv Detail & Related papers (2022-09-22T20:29:49Z) - MATS: An Interpretable Trajectory Forecasting Representation for
Planning and Control [46.86174832000696]
Reasoning about human motion is a core component of modern human-robot interactive systems.
One of the main uses of behavior prediction in autonomous systems is to inform robot motion planning and control.
We propose a new output representation for trajectory forecasting that is more amenable to downstream planning and control use.
arXiv Detail & Related papers (2020-09-16T07:32:37Z) - Long-Horizon Visual Planning with Goal-Conditioned Hierarchical
Predictors [124.30562402952319]
The ability to predict and plan into the future is fundamental for agents acting in the world.
Current learning approaches for visual prediction and planning fail on long-horizon tasks.
We propose a framework for visual prediction and planning that is able to overcome both of these limitations.
arXiv Detail & Related papers (2020-06-23T17:58:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.