Compositional Semantic Parsing with Large Language Models

• URL: http://arxiv.org/abs/2209.15003v2
• Date: Fri, 30 Sep 2022 01:15:10 GMT
• Title: Compositional Semantic Parsing with Large Language Models
• Authors: Andrew Drozdov, Nathanael Sch\"arli, Ekin Aky\"urek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, Denny Zhou
• Abstract summary: We identify challenges in more realistic semantic parsing tasks with larger vocabulary. Our best method is based on least-to-most prompting. We expect similar efforts will lead to new results in other tasks and domains.
• Score: 27.627684573915147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans can reason compositionally when presented with new tasks. Previous research shows that appropriate prompting techniques enable large language models (LLMs) to solve artificial compositional generalization tasks such as SCAN. In this work, we identify additional challenges in more realistic semantic parsing tasks with larger vocabulary and refine these prompting techniques to address them. Our best method is based on least-to-most prompting: it decomposes the problem using prompting-based syntactic parsing, then uses this decomposition to select appropriate exemplars and to sequentially generate the semantic parse. This method allows us to set a new state of the art for CFQ while requiring only 1% of the training data used by traditional approaches. Due to the general nature of our approach, we expect similar efforts will lead to new results in other tasks and domains, especially for knowledge-intensive applications.

Related papers

• Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey [0.0]
  We present a literature survey geared at recent advances in analysis, methods, and evaluation schemes for compositional generalization. This type of generalization is particularly relevant to the semantic parsing community for applications such as task-oriented dialogue.
  arXiv  Detail & Related papers  (2024-04-15T10:44:58Z)
• Get the gist? Using large language models for few-shot decontextualization [3.2271521193192734]
  We propose a few-shot method of decontextualization using a large language model. We show that this method achieves viable performance on multiple domains using only a small set of examples.
  arXiv  Detail & Related papers  (2023-10-10T02:00:00Z)
• Large Language Models as Analogical Reasoners [155.9617224350088]
  Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks. We introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models.
  arXiv  Detail & Related papers  (2023-10-03T00:57:26Z)
• ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis [54.18659323181771]
  We characterize several different forms of compositional generalization that are desirable in program synthesis. We propose ExeDec, a novel decomposition-based strategy that predicts execution subgoals to solve problems step-by-step informed by program execution at each step.
  arXiv  Detail & Related papers  (2023-07-26T01:07:52Z)
• Conjunct Resolution in the Face of Verbal Omissions [51.220650412095665]
  We propose a conjunct resolution task that operates directly on the text and makes use of a split-and-rephrase paradigm in order to recover the missing elements in the coordination structure. We curate a large dataset, containing over 10K examples of naturally-occurring verbal omissions with crowd-sourced annotations. We train various neural baselines for this task, and show that while our best method obtains decent performance, it leaves ample space for improvement.
  arXiv  Detail & Related papers  (2023-05-26T08:44:02Z)
• On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex [48.588772371355816]
  This paper presents the first empirical study on the adversarial robustness of a large prompt-based language model of code, codex. Our results demonstrate that the state-of-the-art (SOTA) code-language models are vulnerable to carefully crafted adversarial examples.
  arXiv  Detail & Related papers  (2023-01-30T13:21:00Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• Training Naturalized Semantic Parsers with Very Little Data [10.709587018625275]
  State-of-the-art (SOTA) semantics are seq2seq architectures based on large language models that have been pretrained on vast amounts of text. Recent work has explored a reformulation of semantic parsing whereby the output sequences are themselves natural language sentences. We show that this method delivers new SOTA few-shot performance on the Overnight dataset.
  arXiv  Detail & Related papers  (2022-04-29T17:14:54Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.