Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study

• URL: http://arxiv.org/abs/2607.03174v1
• Date: Fri, 03 Jul 2026 10:13:59 GMT
• Title: Effectiveness of LLM-based Software Diversity for Reliability Improvement -- an Empirical Study
• Abstract summary: Traditional software diversity is costly to obtain, since it requires multiple implementations as well as the corresponding validation, maintenance, and deployment effort.<n>Recent advances in Large Language Models (LLMs) may change this.<n>LLMs produce many candidate implementations of the same specification quickly, across different models, decoding settings, and programming languages.
• Score: 2.6342357827983
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Software diversity has been extensively studied as a means of reducing the risk of common-mode failures. Classic work showed that the central issue is whether failures of diversely redundant components overlap in ways that limit the reliability gains. Traditional software diversity is costly to obtain, since it requires multiple implementations as well as the corresponding validation, maintenance, and deployment effort. Recent advances in Large Language Models (LLMs) may change this. LLMs enable inexpensive code generation: they produce many candidate implementations of the same specification quickly, across different models, decoding settings, and programming languages. This raises a natural question: can LLMs serve as practical generators of software diversity, and how much reliability improvement can that diversity actually provide? In this paper, we extend classical empirical studies of software diversity in human-written programs to LLM-generated code. We study three specifications using both historical human-written programs and large pools of LLM-generated ones evaluated under a common compilation, sandboxing, and exhaustive test suite. We explore LLM diversity along multiple axes, including model family, generation temperature, and programming language. Reliability improvement is evaluated in a 1-out-of-2 configuration across both homogeneous and heterogeneous program populations, including within-LLM pairings and pairings across programming languages and across LLM-generated and human-written programs. The results show that combining LLM-generated programs, especially in heterogeneous settings, can yield reliability gains, although this is partly conditioned by the programming language and generation setting. Taken together, these findings suggest that LLMs provide a scalable source of comparatively low-cost programs whose diversity can be leveraged for reliability improvement.

Related papers

• Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development [3.4912596012054316]
  Neural Source Code Summarisation (NSCS) aims to generate natural language summaries of source code to improve developers' and maintainers' understanding of code.<n>We show how combining Transformer-generated summaries in prompt engineering may enable LLMs to create better source code summaries.
  arXiv  Detail & Related papers  (2026-07-23T05:27:24Z)
• Code Fingerprints: Disentangled Attribution of LLM-Generated Code [7.515488307576106]
  We study the problem of model-level code attribution, which aims to determine the source LLM responsible for generated code.<n>We propose the Disentangled Code Attribution Network (DCAN), which separates Source-Agnostic semantic information from Source-Specific stylistic representations.<n>We construct the first large-scale benchmark dataset comprising code generated by four widely used Large Language Models (LLMs) across four programming languages.
  arXiv  Detail & Related papers  (2026-03-04T15:58:36Z)
• SLMFix: Leveraging Small Language Models for Error Fixing with Reinforcement Learning [39.94602104823846]
  Large language models (LLMs) generate programs that contains syntactic errors and fail to complete the given tasks.<n>In this work, we propose SLMFix, a novel code generation pipeline that leverages a small language model (SLM) finetuned using reinforcement learning (RL) techniques.
  arXiv  Detail & Related papers  (2025-11-24T18:56:47Z)
• CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings [32.72039589832989]
  Large language models (LLMs) have revolutionized code generation, automating programming with remarkable efficiency.<n>These advancements challenge programming skills, ethics, and assessment integrity, making the detection of LLM-generated code essential for maintaining accountability and standards.<n>We propose a framework capable of distinguishing between human- and LLM-written code across multiple programming languages, code generators, and domains.
  arXiv  Detail & Related papers  (2025-03-17T21:41:37Z)
• When One LLM Drools, Multi-LLM Collaboration Rules [98.71562711695991]
  We argue for multi-LLM collaboration to better represent the extensive diversity of data, skills, and people.<n>We organize existing multi-LLM collaboration methods into a hierarchy, based on the level of access and information exchange.<n>We envision multi-LLM collaboration as an essential path toward compositional intelligence and collaborative AI development.
  arXiv  Detail & Related papers  (2025-02-06T21:13:44Z)
• Multi-Programming Language Sandbox for LLMs [78.99934332554963]
  out-of-the-box multi-programming language sandbox designed to provide unified and comprehensive feedback from compiler and analysis tools for Large Language Models (LLMs) It can automatically identify the programming language of the code, compiling and executing it within an isolated sub-sandbox to ensure safety and stability.
  arXiv  Detail & Related papers  (2024-10-30T14:46:43Z)
• Multi-Programming Language Ensemble for Code Generation in Large Language Model [5.882816711878273]
  Large language models (LLMs) have significantly improved code generation, particularly in one-pass code generation. Most existing approaches focus solely on generating code in a single programming language, overlooking the potential of leveraging the multi-language capabilities of LLMs. We propose Multi-Programming Language Ensemble (MPLE), a novel ensemble-based method that utilizes code generation across multiple programming languages to enhance overall performance.
  arXiv  Detail & Related papers  (2024-09-06T08:31:18Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [92.62952504133926]
  This study evaluated the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks.<n>We developed a taxonomy of bugs for incorrect codes and analyzed the root cause for common bug types.<n>We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Knowledge Fusion of Large Language Models [73.28202188100646]
  This paper introduces the notion of knowledge fusion for large language models (LLMs) We externalize their collective knowledge and unique strengths, thereby elevating the capabilities of the target model beyond those of any individual source LLM. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation.
  arXiv  Detail & Related papers  (2024-01-19T05:02:46Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
  We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-02-16T18:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.