HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation

• URL: http://arxiv.org/abs/2505.16978v2
• Date: Sun, 01 Jun 2025 12:49:41 GMT
• Title: HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation
• Authors: Weizhi Tang, Yixuan Li, Chris Sypherd, Elizabeth Polgreen, Vaishak Belle,
• Abstract summary: We study and improve the ability of large language models (LLMs) for few-shot grammar generation.<n>Our findings reveal that existing LLMs perform sub-optimally in grammar generation.<n>We propose an LLM-driven hybrid genetic algorithm, namely HyGenar, to optimize grammar generation.
• Score: 17.367256030047123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grammar plays a critical role in natural language processing and text/code generation by enabling the definition of syntax, the creation of parsers, and guiding structured outputs. Although large language models (LLMs) demonstrate impressive capabilities across domains, their ability to infer and generate grammars has not yet been thoroughly explored. In this paper, we aim to study and improve the ability of LLMs for few-shot grammar generation, where grammars are inferred from sets of a small number of positive and negative examples and generated in Backus-Naur Form. To explore this, we introduced a novel dataset comprising 540 structured grammar generation challenges, devised 6 metrics, and evaluated 8 various LLMs against it. Our findings reveal that existing LLMs perform sub-optimally in grammar generation. To address this, we propose an LLM-driven hybrid genetic algorithm, namely HyGenar, to optimize grammar generation. HyGenar achieves substantial improvements in both the syntactic and semantic correctness of generated grammars across LLMs.

Related papers

• RELIC: Evaluating Compositional Instruction Following via Language Recognition [37.49115450182637]
  Large language models (LLMs) are increasingly expected to perform tasks based only on a specification of the task provided in context.<n>We introduce the Recognition of Languages In-Context (RELIC) framework to evaluate instruction following using language recognition.
  arXiv  Detail & Related papers  (2025-06-05T16:17:24Z)
• GEM: Empowering LLM for both Embedding Generation and Language Understanding [11.081595808236239]
  We propose Generative Embedding large language Model (GEM) to generate high-quality text embeddings.<n>Our method inserts new special token(s) into a text body, and generates summarization embedding of the text by manipulating the attention mask.<n>Our results indicate that our approach can empower LLMs with state-of-the-art text embedding capabilities while maintaining their original NLP performance.
  arXiv  Detail & Related papers  (2025-06-04T18:02:07Z)
• Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs? [29.690921649662744]
  Grammar serves as a cornerstone in programming languages and software engineering.<n>Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models.<n>We develop a series of billion-scale GrammarCoder models, incorporating grammar rules in the code generation process.
  arXiv  Detail & Related papers  (2025-03-07T15:23:13Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs)<n>We find that fine-tuning text embedding models on LLM-generated texts yields excellent classification accuracy.<n>We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking [5.174301428591665]
  Large Language Models (LLMs) are widely used for tasks such as natural language and code generation.<n>Their outputs often suffer from issues like hallucination, toxicity, and incorrect results.<n>Current libraries for structured LLM generation rely on left-to-right decoding without support for backtracking.<n>IterGen enables users to move both forward and backward within the generated output based on grammar symbols.
  arXiv  Detail & Related papers  (2024-10-09T16:21:38Z)
• Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models [59.60208063956459]
  Large Language Models (LLMs) require high quality instruction data for effective alignment.<n>We present Genetic-Instruct, a scalable algorithm for synthesizing large-scale, high quality coding instructions.
  arXiv  Detail & Related papers  (2024-07-29T20:42:59Z)
• ReMoDetect: Reward Models Recognize Aligned LLM's Generations [55.06804460642062]
  Large language models (LLMs) generate human-preferable texts. In this paper, we identify the common characteristics shared by these models. We propose two training schemes to further improve the detection ability of the reward model.
  arXiv  Detail & Related papers  (2024-05-27T17:38:33Z)
• Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation [128.01050030936028]
  We propose an information refinement training method named InFO-RAG. InFO-RAG is low-cost and general across various tasks. It improves the performance of LLaMA2 by an average of 9.39% relative points.
  arXiv  Detail & Related papers  (2024-02-28T08:24:38Z)
• AlignedCoT: Prompting Large Language Models via Native-Speaking Demonstrations [52.43593893122206]
  Alignedcot is an in-context learning technique for invoking Large Language Models. It achieves consistent and correct step-wise prompts in zero-shot scenarios. We conduct experiments on mathematical reasoning and commonsense reasoning.
  arXiv  Detail & Related papers  (2023-11-22T17:24:21Z)
• Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models [49.74036826946397]
  This study investigates constrained text generation for large language models (LLMs) Our research mainly focuses on mainstream open-source LLMs, categorizing constraints into lexical, structural, and relation-based types. Results illuminate LLMs' capacity and deficiency to incorporate constraints and provide insights for future developments in constrained text generation.
  arXiv  Detail & Related papers  (2023-10-25T03:58:49Z)
• Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning [57.74233319453229]
  Large language models (LLMs) have emerged as a groundbreaking technology and their unparalleled text generation capabilities have sparked interest in their application to the fundamental sentence representation learning task. We propose MultiCSR, a multi-level contrastive sentence representation learning framework that decomposes the process of prompting LLMs to generate a corpus. Our experiments reveal that MultiCSR enables a less advanced LLM to surpass the performance of ChatGPT, while applying it to ChatGPT achieves better state-of-the-art results.
  arXiv  Detail & Related papers  (2023-10-17T03:21:43Z)
• Grammar Prompting for Domain-Specific Language Generation with Large Language Models [40.831045850285776]
  Large language models (LLMs) can learn to perform a wide range of natural language tasks from just a handful of in-context examples. We propose emphgrammar prompting, a simple approach to enable LLMs to use external knowledge and domain-specific constraints.
  arXiv  Detail & Related papers  (2023-05-30T17:26:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.