Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey

• URL: http://arxiv.org/abs/2404.13074v1
• Date: Mon, 15 Apr 2024 10:44:58 GMT
• Title: Towards Compositionally Generalizable Semantic Parsing in Large Language Models: A Survey
• Authors: Amogh Mannekote,
• Abstract summary: We present a literature survey geared at recent advances in analysis, methods, and evaluation schemes for compositional generalization. This type of generalization is particularly relevant to the semantic parsing community for applications such as task-oriented dialogue.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Compositional generalization is the ability of a model to generalize to complex, previously unseen types of combinations of entities from just having seen the primitives. This type of generalization is particularly relevant to the semantic parsing community for applications such as task-oriented dialogue, text-to-SQL parsing, and information retrieval, as they can harbor infinite complexity. Despite the success of large language models (LLMs) in a wide range of NLP tasks, unlocking perfect compositional generalization still remains one of the few last unsolved frontiers. The past few years has seen a surge of interest in works that explore the limitations of, methods to improve, and evaluation metrics for compositional generalization capabilities of LLMs for semantic parsing tasks. In this work, we present a literature survey geared at synthesizing recent advances in analysis, methods, and evaluation schemes to offer a starting point for both practitioners and researchers in this area.

Related papers

• SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation [21.68354181391989]
  We propose SPOR, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation. We demonstrate SPOR on two different datasets and evaluate some existing language models including LLMs.
  arXiv  Detail & Related papers  (2024-05-17T09:25:30Z)
• Can LLM find the green circle? Investigation and Human-guided tool manipulation for compositional generalization [28.069928613367978]
  Large language models (LLMs) exhibit impressive generalization abilities on many tasks through in-context learning (ICL) We propose a human-guided tool manipulation framework (HTM) that generates tools for sub-questions and integrates multiple tools. Experiments show that our method achieves state-of-the-art performance on two compositional generalization benchmarks and outperforms existing methods on the most challenging test split by 70%.
  arXiv  Detail & Related papers  (2023-12-12T22:11:17Z)
• In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax [36.98247762224868]
  In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks. Do models infer the underlying structure of the task defined by the context, or do they rely on superficial generalizations that only generalize to identically distributed examples? In experiments with models from the GPT, PaLM, and Llama 2 families, we find large variance across LMs. The variance is explained more by the composition of the pre-training corpus and supervision methods than by model size.
  arXiv  Detail & Related papers  (2023-11-13T23:52:43Z)
• Large Search Model: Redefining Search Stack in the Era of LLMs [63.503320030117145]
  We introduce a novel conceptual framework called large search model, which redefines the conventional search stack by unifying search tasks with one large language model (LLM) All tasks are formulated as autoregressive text generation problems, allowing for the customization of tasks through the use of natural language prompts. This proposed framework capitalizes on the strong language understanding and reasoning capabilities of LLMs, offering the potential to enhance search result quality while simultaneously simplifying the existing cumbersome search stack.
  arXiv  Detail & Related papers  (2023-10-23T05:52:09Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• ExeDec: Execution Decomposition for Compositional Generalization in Neural Program Synthesis [54.18659323181771]
  We characterize several different forms of compositional generalization that are desirable in program synthesis. We propose ExeDec, a novel decomposition-based strategy that predicts execution subgoals to solve problems step-by-step informed by program execution at each step.
  arXiv  Detail & Related papers  (2023-07-26T01:07:52Z)
• Compositional Semantic Parsing with Large Language Models [27.627684573915147]
  We identify challenges in more realistic semantic parsing tasks with larger vocabulary. Our best method is based on least-to-most prompting. We expect similar efforts will lead to new results in other tasks and domains.
  arXiv  Detail & Related papers  (2022-09-29T17:58:28Z)
• Compositional Generalization Requires Compositional Parsers [69.77216620997305]
  We compare sequence-to-sequence models and models guided by compositional principles on the recent COGS corpus. We show structural generalization is a key measure of compositional generalization and requires models that are aware of complex structure.
  arXiv  Detail & Related papers  (2022-02-24T07:36:35Z)
• Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge [59.22170796793179]
  Transformers Language Models (TLMs) were tested on a benchmark for the textitdynamic estimation of thematic fit Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge.
  arXiv  Detail & Related papers  (2021-07-22T20:52:26Z)
• Improving Compositional Generalization in Semantic Parsing [54.4720965813889]
  Generalization of models to out-of-distribution (OOD) data has captured tremendous attention recently. We investigate compositional generalization in semantic parsing, a natural test-bed for compositional generalization.
  arXiv  Detail & Related papers  (2020-10-12T12:34:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.