Binding Language Models in Symbolic Languages

• URL: http://arxiv.org/abs/2210.02875v1
• Date: Thu, 6 Oct 2022 12:55:17 GMT
• Title: Binding Language Models in Symbolic Languages
• Authors: Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, Tao Yu
• Abstract summary: Binder is a training-free neural-symbolic framework that maps the task input to a program. In the parsing stage, Codex is able to identify the part of the task input that cannot be answerable by the original programming language. In the execution stage, Codex can perform versatile functionalities given proper prompts in the API calls.
• Score: 146.3027328556881
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Though end-to-end neural approaches have recently been dominating NLP tasks in both performance and ease-of-use, they lack interpretability and robustness. We propose Binder, a training-free neural-symbolic framework that maps the task input to a program, which (1) allows binding a unified API of language model (LM) functionalities to a programming language (e.g., SQL, Python) to extend its grammar coverage and thus tackle more diverse questions, (2) adopts an LM as both the program parser and the underlying model called by the API during execution, and (3) requires only a few in-context exemplar annotations. Specifically, we employ GPT-3 Codex as the LM. In the parsing stage, with only a few in-context exemplars, Codex is able to identify the part of the task input that cannot be answerable by the original programming language, correctly generate API calls to prompt Codex to solve the unanswerable part, and identify where to place the API calls while being compatible with the original grammar. In the execution stage, Codex can perform versatile functionalities (e.g., commonsense QA, information extraction) given proper prompts in the API calls. Binder achieves state-of-the-art results on WikiTableQuestions and TabFact datasets, with explicit output programs that benefit human debugging. Note that previous best systems are all finetuned on tens of thousands of task-specific samples, while Binder only uses dozens of annotations as in-context exemplars without any training. Our code is available at https://github.com/HKUNLP/Binder .

Related papers

• AppBench: Planning of Multiple APIs from Various APPs for Complex User Instruction [24.67142048995415]
  Large Language Models (LLMs) can interact with the real world by connecting with versatile external APIs. We introduce textttAppBench, the first benchmark to evaluate LLMs' ability to plan and execute multiple APIs from various sources.
  arXiv  Detail & Related papers  (2024-10-10T04:03:13Z)
• Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learning [14.351476383642016]
  We propose a novel approach, named Code2API, to automatically perform APIzation for Stack Overflow code snippets. Code2API does not require additional model training or any manual crafting rules. It can be easily deployed on personal computers without relying on other external tools.
  arXiv  Detail & Related papers  (2024-05-06T14:22:17Z)
• AskIt: Unified Programming Interface for Programming with Large Language Models [0.0]
  Large Language Models (LLMs) exhibit a unique phenomenon known as emergent abilities, demonstrating adeptness across numerous tasks. This paper introduces AskIt, a domain-specific language specifically designed for LLMs. Across 50 tasks, AskIt generated concise prompts, achieving a 16.14 % reduction in prompt length compared to benchmarks.
  arXiv  Detail & Related papers  (2023-08-29T21:44:27Z)
• SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding [103.34092301324425]
  Large language models (LLMs) have shown impressive ability for open-domain NLP tasks. We present SeqGPT, a bilingual (i.e., English and Chinese) open-source autoregressive model specially enhanced for open-domain natural language understanding.
  arXiv  Detail & Related papers  (2023-08-21T07:31:19Z)
• Private-Library-Oriented Code Generation with Large Language Models [52.73999698194344]
  This paper focuses on utilizing large language models (LLMs) for code generation in private libraries. We propose a novel framework that emulates the process of programmers writing private code. We create four private library benchmarks, including TorchDataEval, TorchDataComplexEval, MonkeyEval, and BeatNumEval.
  arXiv  Detail & Related papers  (2023-07-28T07:43:13Z)
• DocCoder: Generating Code by Retrieving and Reading Docs [87.88474546826913]
  We introduce DocCoder, an approach that explicitly leverages code manuals and documentation. Our approach is general, can be applied to any programming language, and is agnostic to the underlying neural model.
  arXiv  Detail & Related papers  (2022-07-13T06:47:51Z)
• On the Effectiveness of Pretrained Models for API Learning [8.788509467038743]
  Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc. Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner. Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences.
  arXiv  Detail & Related papers  (2022-04-05T20:33:24Z)
• Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start [125.23550801424328]
  Universal Few-shot textual Entailment (UFO-Entail) We introduce Universal Few-shot textual Entailment (UFO-Entail) We demonstrate that this framework enables a pretrained entailment model to work well on new entailment domains in a few-shot setting.
  arXiv  Detail & Related papers  (2020-10-06T09:50:25Z)
• Language Models are Few-Shot Learners [61.36677350504291]
  We show that scaling up language models greatly improves task-agnostic, few-shot performance. We train GPT-3, an autoregressive language model with 175 billion parameters, and test its performance in the few-shot setting. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
  arXiv  Detail & Related papers  (2020-05-28T17:29:03Z)
• Template Guided Text Generation for Task-Oriented Dialogue [9.690158790639131]
  Virtual assistants such as Google Assistant, Amazon Alexa, and Apple Siri enable users to interact with a large number of services and APIs on the web using natural language. In this work, we investigate two methods for Natural Language Generation using a single domain-independent model across a large number of APIs.
  arXiv  Detail & Related papers  (2020-04-30T17:51:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.