Generative transformations and patterns in LLM-native approaches for software verification and falsification

• URL: http://arxiv.org/abs/2404.09384v3
• Date: Mon, 06 Oct 2025 21:35:53 GMT
• Title: Generative transformations and patterns in LLM-native approaches for software verification and falsification
• Authors: Víctor A. Braberman, Flavia Bonomo-Braberman, Yiannis Charalambous, Juan G. Colonna, Lucas C. Cordeiro, Rosiane de Freitas,
• Abstract summary: We argue that a foundational step towards a more disciplined engineering practice is a systematic understanding of the core functional units-generative transformations.<n>We first present a fine-grained taxonomy of generative transformations, abstracting prompt-based interactions into conceptual signatures.<n>Our analysis not only validates the utility of the taxonomy but also surfaces strategic gaps and cross-dimensional relationships.
• Score: 1.4595796095047369
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The emergence of prompting as the dominant paradigm for leveraging Large Language Models (LLMs) has led to a proliferation of LLM-native software, where application behavior arises from complex, stochastic data transformations. However, the engineering of such systems remains largely exploratory and ad-hoc, hampered by the absence of conceptual frameworks, ex-ante methodologies, design guidelines, and specialized benchmarks. We argue that a foundational step towards a more disciplined engineering practice is a systematic understanding of the core functional units--generative transformations--and their compositional patterns within LLM-native applications. Focusing on the rich domain of software verification and falsification, we conduct a secondary study of over 100 research proposals to address this gap. We first present a fine-grained taxonomy of generative transformations, abstracting prompt-based interactions into conceptual signatures. This taxonomy serves as a scaffolding to identify recurrent transformation relationship patterns--analogous to software design patterns--that characterize solution approaches in the literature. Our analysis not only validates the utility of the taxonomy but also surfaces strategic gaps and cross-dimensional relationships, offering a structured foundation for future research in modular and compositional LLM application design, benchmarking, and the development of reliable LLM-native systems.

Related papers

• Beyond Basic Specifications? A Systematic Study of Logical Constructs in LLM-based Specification Generation [29.231420590756954]
  Large language models (LLMs) for the automatic generation of program specifications has emerged as a promising avenue for enhancing verification efficiency.<n>We propose incorporating logical constructs into existing LLM-based specification generation framework.<n>We conduct an empirical study aimed at exploring the impact of various types of syntactic constructs on specification generation framework.
  arXiv  Detail & Related papers  (2026-01-31T13:19:40Z)
• A Survey of Vibe Coding with Large Language Models [93.88284590533242]
  "Vibe Coding" is a development methodology where developers validate AI-generated implementations through outcome observation.<n>Despite its transformative potential, the effectiveness of this emergent paradigm remains under-explored.<n>This survey provides the first comprehensive and systematic review of Vibe Coding with large language models.
  arXiv  Detail & Related papers  (2025-10-14T11:26:56Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• Designing Empirical Studies on LLM-Based Code Generation: Towards a Reference Framework [0.3568466510804538]
  We propose a theoretical framework for designing and reporting empirical studies on large language models (LLMs)-based code generation.<n>The framework is grounded in both our prior experience conducting such experiments and a comparative analysis of key similarities and differences among recent studies.<n>It organizes evaluation around core components such as problem sources, quality attributes, and metrics, supporting structured and systematic experimentation.
  arXiv  Detail & Related papers  (2025-10-04T16:15:54Z)
• The Landscape of Agentic Reinforcement Learning for LLMs: A Survey [103.32591749156416]
  The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL)<n>This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM-RL with the temporally extended, partially observable Markov decision processes (POMDPs) that define Agentic RL.
  arXiv  Detail & Related papers  (2025-09-02T17:46:26Z)
• A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models [2.518519330408713]
  Large Language Models (LLMs) in software engineering have sparked interest in their use for software vulnerability detection.<n>The rapid development of this field has resulted in a fragmented research landscape.<n>This fragmentation makes it difficult to obtain a clear overview of the state-of-the-art or compare and categorize studies meaningfully.
  arXiv  Detail & Related papers  (2025-07-30T13:17:16Z)
• Integrating LLM in Agent-Based Social Simulation: Opportunities and Challenges [0.7739037410679168]
  The paper reviews recent findings on the ability of Large Language Models to replicate key aspects of human cognition.<n>The second part surveys emerging applications of LLMs in multi-agent simulation frameworks.<n>The paper concludes by advocating for hybrid approaches that integrate LLMs into traditional agent-based modeling platforms.
  arXiv  Detail & Related papers  (2025-07-25T15:15:35Z)
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey [69.45421620616486]
  This work presents the first structured taxonomy and analysis of discrete tokenization methods designed for large language models (LLMs)<n>We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines.<n>We identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints.
  arXiv  Detail & Related papers  (2025-07-21T10:52:14Z)
• A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems [93.8285345915925]
  Reasoning is a fundamental cognitive process that enables logical inference, problem-solving, and decision-making. With the rapid advancement of large language models (LLMs), reasoning has emerged as a key capability that distinguishes advanced AI systems. We categorize existing methods along two dimensions: (1) Regimes, which define the stage at which reasoning is achieved; and (2) Architectures, which determine the components involved in the reasoning process.
  arXiv  Detail & Related papers  (2025-04-12T01:27:49Z)
• Aligning Multimodal LLM with Human Preference: A Survey [62.89722942008262]
  Large language models (LLMs) can handle a wide variety of general tasks with simple prompts, without the need for task-specific training. Multimodal Large Language Models (MLLMs) have demonstrated impressive potential in tackling complex tasks involving visual, auditory, and textual data. However, critical issues related to truthfulness, safety, o1-like reasoning, and alignment with human preference remain insufficiently addressed.
  arXiv  Detail & Related papers  (2025-03-18T17:59:56Z)
• A Survey on Post-training of Large Language Models [185.51013463503946]
  Large Language Models (LLMs) have fundamentally transformed natural language processing, making them indispensable across domains ranging from conversational systems to scientific exploration.<n>These challenges necessitate advanced post-training language models (PoLMs) to address shortcomings, such as restricted reasoning capacities, ethical uncertainties, and suboptimal domain-specific performance.<n>This paper presents the first comprehensive survey of PoLMs, systematically tracing their evolution across five core paradigms: Fine-tuning, which enhances task-specific accuracy; Alignment, which ensures ethical coherence and alignment with human preferences; Reasoning, which advances multi-step inference despite challenges in reward design; Integration and Adaptation, which
  arXiv  Detail & Related papers  (2025-03-08T05:41:42Z)
• Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy [14.041979999979166]
  Large Language Models (LLMs) and Multi-Agent LLMs (MALLMs) introduce non-determinism unlike traditional or machine learning software. This paper presents a taxonomy for LLM test case design, informed by both the research literature, our experience, and open-source tools that represent the state of practice.
  arXiv  Detail & Related papers  (2025-03-01T13:15:56Z)
• A Survey of Model Architectures in Information Retrieval [59.61734783818073]
  The period from 2019 to the present has represented one of the biggest paradigm shifts in information retrieval (IR) and natural language processing (NLP)<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude with a forward-looking discussion of emerging challenges and future directions.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights [12.424610893030353]
  Large Language Models (LLMs) are emerging as transformative tools for software vulnerability detection. This paper provides a detailed survey of LLMs in vulnerability detection. We address challenges such as cross-language vulnerability detection, multimodal data integration, and repository-level analysis.
  arXiv  Detail & Related papers  (2025-02-10T21:33:38Z)
• Towards a Probabilistic Framework for Analyzing and Improving LLM-Enabled Software [0.0]
  Large language model (LLM)-enabled systems are a significant challenge in software engineering.<n>We propose a probabilistic framework for systematically analyzing and improving these systems.
  arXiv  Detail & Related papers  (2025-01-10T22:42:06Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Large Language Models for Anomaly and Out-of-Distribution Detection: A Survey [18.570066068280212]
  Large Language Models (LLMs) have demonstrated their effectiveness not only in natural language processing but also in broader applications. This survey focuses on the problem of anomaly and OOD detection under the context of LLMs. We propose a new taxonomy to categorize existing approaches into two classes based on the role played by LLMs.
  arXiv  Detail & Related papers  (2024-09-03T15:22:41Z)
• A Software Engineering Perspective on Testing Large Language Models: Research, Practice, Tools and Benchmarks [2.8061460833143346]
  Large Language Models (LLMs) are rapidly becoming ubiquitous both as stand-alone tools and as components of current and future software systems. To enable usage of LLMs in the high-stake or safety-critical systems of 2030, they need to undergo rigorous testing.
  arXiv  Detail & Related papers  (2024-06-12T13:45:45Z)
• Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling [0.9095496510579351]
  We investigate the untapped potential of large language models (LLMs) as an alternative for uncovering the underlying topics within extensive text corpora. Our findings indicate that LLMs with appropriate prompts can stand out as a viable alternative, capable of generating relevant topic titles and adhering to human guidelines to refine and merge topics.
  arXiv  Detail & Related papers  (2024-03-24T17:39:51Z)
• A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications [11.568575664316143]
  This paper provides a structured overview of recent advancements in prompt engineering, categorized by application area. We provide a summary detailing the prompting methodology, its applications, the models involved, and the datasets utilized. This systematic analysis enables a better understanding of this rapidly developing field and facilitates future research by illuminating open challenges and opportunities for prompt engineering.
  arXiv  Detail & Related papers  (2024-02-05T19:49:13Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation [92.43001160060376]
  We study the factuality of large language models (LLMs) in the context of answering questions that test current world knowledge. We introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA.
  arXiv  Detail & Related papers  (2023-10-05T00:04:12Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Recent Advances in Direct Speech-to-text Translation [58.692782919570845]
  We categorize the existing research work into three directions based on the main challenges -- modeling burden, data scarcity, and application issues. For the challenge of data scarcity, recent work resorts to many sophisticated techniques, such as data augmentation, pre-training, knowledge distillation, and multilingual modeling. We analyze and summarize the application issues, which include real-time, segmentation, named entity, gender bias, and code-switching.
  arXiv  Detail & Related papers  (2023-06-20T16:14:27Z)
• LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models [67.19124099815645]
  We propose a novel Language-Aware Soft Prompting (LASP) learning method to alleviate base class overfitting. LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available. LASP matches and surpasses, for the first time, the accuracy on novel classes obtained by hand-crafted prompts and CLIP for 8 out of 11 test datasets.
  arXiv  Detail & Related papers  (2022-10-03T17:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.