Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval

• URL: http://arxiv.org/abs/2603.22717v1
• Date: Tue, 24 Mar 2026 02:13:31 GMT
• Title: Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval
• Abstract summary: Multi-LLMSECCODEEVAL is a framework for assessing and enhancing security across the vulnerability management lifecycle.<n>We benchmark ten pipelines spanning single model, ensemble, collaborative, and hybrid designs.
• Score: 22.812621042022105
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatically generating source code from natural language using large language models (LLMs) is becoming common, yet security vulnerabilities persist despite advances in fine tuning and prompting. In this work, we systematically evaluate whether multi LLM ensembles and collaborative strategies can meaningfully improve secure code generation. We present MULTI-LLMSECCODEEVAL, a framework for assessing and enhancing security across the vulnerability management lifecycle by combining multiple LLMs with static analysis and structured collaboration. Using SecLLMEval and SecLLMHolmes, we benchmark ten pipelines spanning single model, ensemble, collaborative, and hybrid designs. Our results show that ensemble pipelines augmented with static analysis improve secure code generation over single LLM baselines by up to 47.3% on SecLLMEval and 19.3% on SecLLMHolmes, while purely LLM based collaborative pipelines yield smaller gains of 8.9% to 22.3%. Hybrid pipelines that integrate ensembling, detection, and patching achieve the strongest security performance, outperforming the best ensemble baseline by 1.78% to 4.72% and collaborative baselines by 19.81% to 26.78%. Ablation studies reveal that model scale alone does not ensure security. Smaller, structured multi model ensembles consistently outperform large monolithic LLMs. Overall, our findings demonstrate that secure code does not emerge from scale, but from carefully orchestrated multi model system design.

Related papers

• Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation [0.0]
  Cloud misconfiguration remains a leading cause of security incidents, yet whether LLMs and SLMs can generate security-compliant Infrastructure-as-Code is an open question.<n>We benchmark seven models, three closed LLMs (Claude Opus 4, GPT-5.4, Gemini 2.5 Pro) and four open SLMs (Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, Magicoder-S-CL-7B) on AWS Terraform generation across 17 scenarios, integrating Checkov and Trivy scanners into a GitLab CI/CD pipeline and evaluating two prompt strategies at three security levels (
  arXiv  Detail & Related papers  (2026-08-02T14:45:12Z)
• Robust Multimodal Safety via Conditional Decoding [52.92816441364308]
  Multimodal large-language models (MLLMs) often experience degraded safety alignment when harmful queries exploit cross-modal interactions.<n>We propose a simple conditional decoding strategy, CASA, that utilizes internal representations of MLLMs to predict a binary safety token before response generation.
  arXiv  Detail & Related papers  (2026-03-31T23:19:50Z)
• RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories [58.32028251925354]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, but their proficiency in producing secure code remains a critical, under-explored area.<n>We introduce RealSec-bench, a new benchmark for secure code generation meticulously constructed from real-world, high-risk Java repositories.
  arXiv  Detail & Related papers  (2026-01-30T08:29:01Z)
• Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback [1.1017250479834206]
  Large Language Models (LLMs) can generate code but often introduce security vulnerabilities, logical inconsistencies, and compilation errors.<n>We propose a retrieval-augmented, multi-tool repair workflow in which a single code-generating LLM iteratively refines its outputs.
  arXiv  Detail & Related papers  (2026-01-01T23:34:00Z)
• Latent Collaboration in Multi-Agent Systems [140.51506923969345]
  Multi-agent systems (MAS) extend from independent single-model reasoning to coordinative system-level intelligence.<n>We introduce LatentMAS, an end-to-end training-free framework that enables pure latent collaboration among LLM agents.
  arXiv  Detail & Related papers  (2025-11-25T18:56:57Z)
• Benchmarking Correctness and Security in Multi-Turn Code Generation [41.75392001830794]
  We introduce MTSec, the first benchmark to evaluate correctness and security in multi-turn coding scenarios.<n>We evaluate 32 open- and closed-source models, and three agent-scaffolding on MT-Sec.<n>We find that while agent-generated scaffoldings boost single-turn code generation performance, they are not quite as effective in multiturn evaluations.
  arXiv  Detail & Related papers  (2025-10-13T01:20:46Z)
• CTTS: Collective Test-Time Scaling [58.564620942591866]
  Test-time scaling (TTS) has emerged as a promising, training-free approach for enhancing large language model (LLM) performance.<n>We introduce Collective Test-Time Scaling (CTTS) to overcome the dominant single test-time scaling (STTS) paradigm.<n>CTTS-MM is a novel framework that operationalizes multi-agent and multi-reward collaboration.
  arXiv  Detail & Related papers  (2025-08-05T11:19:08Z)
• FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data [60.09659670497899]
  Large language models (LLMs) have a diverse landscape of models, each excelling at different tasks.<n>This diversity drives researchers to employ multiple LLMs in practice, leaving behind valuable multi-LLM log data.<n>We argue that practical fusion must meet two essential requirements: (1) compatibility with real-world serving scenarios (e.g., local and API-based serving), and (2) flexibility to operate at different stages of the LLM pipeline to meet varied user needs.
  arXiv  Detail & Related papers  (2025-07-14T17:58:02Z)
• A Weighted Byzantine Fault Tolerance Consensus Driven Trusted Multiple Large Language Models Network [53.37983409425452]
  Large Language Models (LLMs) have achieved remarkable success across a wide range of applications.<n>Recently, collaborative frameworks such as the Multi-LLM Network (MultiLLMN) have been introduced.<n>We propose a novel Trusted MultiLLMN framework driven by a weighted Byzantine Fault Tolerance (WBFT) blockchain consensus mechanism.
  arXiv  Detail & Related papers  (2025-05-08T10:04:41Z)
• Collab: Controlled Decoding using Mixture of Agents for LLM Alignment [90.6117569025754]
  Reinforcement learning from human feedback has emerged as an effective technique to align Large Language models.<n>Controlled Decoding provides a mechanism for aligning a model at inference time without retraining.<n>We propose a mixture of agent-based decoding strategies leveraging the existing off-the-shelf aligned LLM policies.
  arXiv  Detail & Related papers  (2025-03-27T17:34:25Z)
• Enhancing LLM Code Generation with Ensembles: A Similarity-Based Selection Approach [6.93983229112122]
  We propose an ensemble approach for large language models (LLMs) in code generation.<n>For voting, we compute syntactic and semantic similarity using CodeBLEU and behavioral equivalence.<n>We show through experiments that our ensemble approach consistently outperforms standalone LLMs.
  arXiv  Detail & Related papers  (2025-03-20T04:38:56Z)
• STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysis [8.059606338318538]
  This work performs the first study on the static analysis integration in LLM-based repository-level code completion. We first implement a framework STALL+, which supports an extendable and customizable integration of multiple static analysis strategies. Our findings show that integrating file-level dependencies in prompting phase performs the best while the integration in post-processing phase performs the worse.
  arXiv  Detail & Related papers  (2024-06-14T13:28:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.