"How to best say it?" : Translating Directives in Machine Language into
Natural Language in the Blocks World
- URL: http://arxiv.org/abs/2107.06886v1
- Date: Wed, 14 Jul 2021 17:59:08 GMT
- Title: "How to best say it?" : Translating Directives in Machine Language into
Natural Language in the Blocks World
- Authors: Sujeong Kim, Amir Tamrakar
- Abstract summary: We propose a method to generate optimal natural language for block placement directives generated by a machine's planner.
We describe an algorithm that progressively and generatively transforms the machine's directive in ECI (Elementary Composable Ideas)-space.
We then define a cost function to evaluate the ease of comprehension of these alternatives and select the best option.
- Score: 7.99536002595393
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a method to generate optimal natural language for block placement
directives generated by a machine's planner during human-agent interactions in
the blocks world. A non user-friendly machine directive, e.g., move(ObjId,
toPos), is transformed into visually and contextually grounded referring
expressions that are much easier for the user to comprehend. We describe an
algorithm that progressively and generatively transforms the machine's
directive in ECI (Elementary Composable Ideas)-space, generating many
alternative versions of the directive. We then define a cost function to
evaluate the ease of comprehension of these alternatives and select the best
option. The parameters for this cost function were derived empirically from a
user study that measured utterance-to-action timings.
Related papers
- Integrating Disambiguation and User Preferences into Large Language Models for Robot Motion Planning [1.9912315834033756]
framework can interpret humans' navigation commands containing temporal elements and translate their natural language instructions into robot motion planning.
We propose methods to resolve the ambiguity in natural language instructions and capture user preferences.
arXiv Detail & Related papers (2024-04-22T19:38:37Z) - DECIDER: A Dual-System Rule-Controllable Decoding Framework for Language Generation [57.07295906718989]
Constrained decoding approaches aim to control the meaning or style of text generated by a Pre-trained Language Model (PLM) using specific target words during inference.
We propose a novel decoding framework, DECIDER, which enables us to program rules on how we complete tasks to control a PLM.
arXiv Detail & Related papers (2024-03-04T11:49:08Z) - Prompt Optimization via Adversarial In-Context Learning [51.18075178593142]
adv-ICL is implemented as a two-player game between a generator and a discriminator.
The generator tries to generate realistic enough output to fool the discriminator.
We show that adv-ICL results in significant improvements over state-of-the-art prompt optimization techniques.
arXiv Detail & Related papers (2023-12-05T09:44:45Z) - Integrating LLMs and Decision Transformers for Language Grounded
Generative Quality-Diversity [0.0]
Quality-Diversity is a branch of optimization that is often applied to problems from the Reinforcement Learning and control domains.
We propose a Large Language Model to augment the repertoire with natural language descriptions of trajectories.
We also propose an LLM-based approach to evaluating the performance of such generative agents.
arXiv Detail & Related papers (2023-08-25T10:00:06Z) - CARTIER: Cartographic lAnguage Reasoning Targeted at Instruction
Execution for Robots [9.393951367344894]
This work explores the capacity of large language models to address problems at the intersection of spatial planning and natural language interfaces for navigation.
We focus on following complex instructions that are more akin to natural conversation than traditional explicit procedural directives typically seen in robotics.
We leverage the 3D simulator AI2Thor to create household query scenarios at scale, and augment it by adding complex language queries for 40 object types.
arXiv Detail & Related papers (2023-07-21T19:09:37Z) - PADL: Language-Directed Physics-Based Character Control [66.517142635815]
We present PADL, which allows users to issue natural language commands for specifying high-level tasks and low-level skills that a character should perform.
We show that our framework can be applied to effectively direct a simulated humanoid character to perform a diverse array of complex motor skills.
arXiv Detail & Related papers (2023-01-31T18:59:22Z) - Correcting Robot Plans with Natural Language Feedback [88.92824527743105]
We explore natural language as an expressive and flexible tool for robot correction.
We show that these transformations enable users to correct goals, update robot motions, and recover from planning errors.
Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.
arXiv Detail & Related papers (2022-04-11T15:22:43Z) - Reshaping Robot Trajectories Using Natural Language Commands: A Study of
Multi-Modal Data Alignment Using Transformers [33.7939079214046]
We provide a flexible language-based interface for human-robot collaboration.
We take advantage of recent advancements in the field of large language models to encode the user command.
We train the model using imitation learning over a dataset containing robot trajectories modified by language commands.
arXiv Detail & Related papers (2022-03-25T01:36:56Z) - Neural Abstructions: Abstractions that Support Construction for Grounded
Language Learning [69.1137074774244]
Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding.
We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model.
We show that with this method a user population is able to build a semantic modification for an open-ended house task in Minecraft.
arXiv Detail & Related papers (2021-07-20T07:01:15Z) - Evaluating the Morphosyntactic Well-formedness of Generated Texts [88.20502652494521]
We propose L'AMBRE -- a metric to evaluate the morphosyntactic well-formedness of text.
We show the effectiveness of our metric on the task of machine translation through a diachronic study of systems translating into morphologically-rich languages.
arXiv Detail & Related papers (2021-03-30T18:02:58Z) - PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping
Pixels to Rewards [40.1007184209417]
We propose a model that maps pixels to rewards, given a free-form natural language description of the task.
Experiments on the Meta-World robot manipulation domain show that language-based rewards significantly improves the sample efficiency of policy learning.
arXiv Detail & Related papers (2020-07-30T15:50:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.