ProTo: Program-Guided Transformer for Program-Guided Tasks
- URL: http://arxiv.org/abs/2110.00804v1
- Date: Sat, 2 Oct 2021 13:46:32 GMT
- Title: ProTo: Program-Guided Transformer for Program-Guided Tasks
- Authors: Zelin Zhao, Karan Samel, Binghong Chen, Le Song
- Abstract summary: We formulate program-guided tasks which require learning to execute a given program on the observed task specification.
We propose the Program-guided Transformer (ProTo), which integrates both semantic and structural guidance of a program.
ProTo executes a program in a learned latent space and enjoys stronger representation ability than previous neural-symbolic approaches.
- Score: 59.34258016795216
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Programs, consisting of semantic and structural information, play an
important role in the communication between humans and agents. Towards learning
general program executors to unify perception, reasoning, and decision making,
we formulate program-guided tasks which require learning to execute a given
program on the observed task specification. Furthermore, we propose the
Program-guided Transformer (ProTo), which integrates both semantic and
structural guidance of a program by leveraging cross-attention and masked
self-attention to pass messages between the specification and routines in the
program. ProTo executes a program in a learned latent space and enjoys stronger
representation ability than previous neural-symbolic approaches. We demonstrate
that ProTo significantly outperforms the previous state-of-the-art methods on
GQA visual reasoning and 2D Minecraft policy learning datasets. Additionally,
ProTo demonstrates better generalization to unseen, complex, and human-written
programs.
Related papers
- A brief overview of programmed instructions for quantum software
education [0.0]
The article presents the programmed instructions method and recent successes in STEM fields before describing its operating mode.
Elements tackled include the core components of programmed instructions, its roots and early use as well as adaptation to complex STEM material.
The aim of this work is to provide high-level guidelines for incorporating programmed instructions in quantum education with the goal of disseminating quantum skills and notions more efficiently to a wider audience.
arXiv Detail & Related papers (2023-12-13T08:47:08Z) - Understanding Programs by Exploiting (Fuzzing) Test Cases [26.8259045248779]
We propose to incorporate the relationship between inputs and possible outputs/behaviors into learning, for achieving a deeper semantic understanding of programs.
To obtain inputs that are representative enough to trigger the execution of most part of the code, we resort to fuzz testing and propose fuzz tuning.
The effectiveness of the proposed method is verified on two program understanding tasks including code clone detection and code classification, and it outperforms current state-of-the-arts by large margins.
arXiv Detail & Related papers (2023-05-23T01:51:46Z) - Hierarchical Programmatic Reinforcement Learning via Learning to Compose
Programs [58.94569213396991]
We propose a hierarchical programmatic reinforcement learning framework to produce program policies.
By learning to compose programs, our proposed framework can produce program policies that describe out-of-distributionally complex behaviors.
The experimental results in the Karel domain show that our proposed framework outperforms baselines.
arXiv Detail & Related papers (2023-01-30T14:50:46Z) - UniRPG: Unified Discrete Reasoning over Table and Text as Program
Generation [32.74302320558048]
We propose UniRPG, a semantic-parsing-based approach advanced in interpretability and scalability.
UniRPG performs unified discrete reasoning over heterogeneous knowledge resources, i.e., table and text, as program generation.
It achieves tremendous improvements and enhances interpretability and scalability compared with state-of-the-art methods.
arXiv Detail & Related papers (2022-10-15T10:17:52Z) - Learning from Self-Sampled Correct and Partially-Correct Programs [96.66452896657991]
We propose to let the model perform sampling during training and learn from both self-sampled fully-correct programs and partially-correct programs.
We show that our use of self-sampled correct and partially-correct programs can benefit learning and help guide the sampling process.
Our proposed method improves the pass@k performance by 3.1% to 12.3% compared to learning from a single reference program with MLE.
arXiv Detail & Related papers (2022-05-28T03:31:07Z) - Procedures as Programs: Hierarchical Control of Situated Agents through
Natural Language [81.73820295186727]
We propose a formalism of procedures as programs, a powerful yet intuitive method of representing hierarchical procedural knowledge for agent command and control.
We instantiate this framework on the IQA and ALFRED datasets for NL instruction following.
arXiv Detail & Related papers (2021-09-16T20:36:21Z) - Learning to Synthesize Programs as Interpretable and Generalizable
Policies [25.258598215642067]
We present a framework that learns to synthesize a program, which details the procedure to solve a task in a flexible and expressive manner.
Experimental results demonstrate that the proposed framework not only learns to reliably synthesize task-solving programs but also outperforms DRL and program synthesis baselines.
arXiv Detail & Related papers (2021-08-31T07:03:06Z) - How could Neural Networks understand Programs? [67.4217527949013]
It is difficult to build a model to better understand programs, by either directly applying off-the-shelf NLP pre-training techniques to the source code, or adding features to the model by theshelf.
We propose a novel program semantics learning paradigm, that the model should learn from information composed of (1) the representations which align well with the fundamental operations in operational semantics, and (2) the information of environment transition.
arXiv Detail & Related papers (2021-05-10T12:21:42Z) - Broader terms curriculum mapping: Using natural language processing and
visual-supported communication to create representative program planning
experiences [62.997667081978825]
Communication difficulties between faculty and non-faculty groups leave unexplored an immense collaboration potential.
This paper presents a method to deliver program plan representations that are universal, self-explanatory, and empowering.
arXiv Detail & Related papers (2021-02-09T13:27:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.