Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

• URL: http://arxiv.org/abs/2606.08840v1
• Date: Sun, 07 Jun 2026 21:10:30 GMT
• Title: Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs
• Abstract summary: Best model reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline.<n>Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage.<n>Gemma-2-27B-IT achieves the highest all-language lint pass rate.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.

Related papers

• M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models [0.0]
  M-GATE is a benchmark of linguistic proficiency spanning 30 typologically diverse languages.<n>We evaluate over 50 models in more than 80 configurations.
  arXiv  Detail & Related papers  (2026-08-04T15:16:05Z)
• AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms [54.99368693313797]
  Existing benchmarks test only individual languages/tools, so the performance numbers are not directly comparable.<n>We address this gap with AlgoVeri, a benchmark that evaluates vericoding of $77$ classical algorithms in Dafny, Verus, and Lean.
  arXiv  Detail & Related papers  (2026-02-10T06:58:26Z)
• Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique [1.5749416770494706]
  Large language models (LLMs) allow for sophisticated qualitative coding of large datasets.<n>We present a simple, generalizable two-stage workflow: an LLM applies a human-designed codebook; a secondary LLM critic performs self-reflection on each positive label.<n>We evaluate this approach on six qualitative codes over 3,000 high-content emails from Apache Software Foundation project evaluation discussions.
  arXiv  Detail & Related papers  (2026-01-14T22:27:13Z)
• Anka: A Domain-Specific Language for Reliable LLM Code Generation [0.0]
  Large Language Models (LLMs) exhibit systematic errors on complex, multi-step programming tasks.<n>We introduce Anka, a domain-specific language () for data transformation pipelines designed with explicit, constrained syntax.<n>Anka achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems.
  arXiv  Detail & Related papers  (2025-12-29T05:28:17Z)
• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• Evaluating Large Language Models for Code Review [2.0261749670612637]
  We tested GPT4o and Gemini 2.0 Flash on 492 AI generated code blocks of varying correctness, along with 164 canonical code blocks from the HumanEval benchmark.<n>With problem descriptions, GPT4o and Gemini 2.0 Flash correctly classified code correctness 68.50% and 63.89% of the time, respectively, and corrected the code 67.83% and 54.26% of the time for the 492 code blocks of varying correctness.
  arXiv  Detail & Related papers  (2025-05-26T16:47:29Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.1875460416205]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages.<n>It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total.<n>Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping [60.458273797431836]
  Decoding by contrasting layers (DoLa) is designed to improve the generation quality of large language models. We find that this approach does not work well on non-English tasks. Inspired by previous interpretability work on language transition during the model's forward pass, we propose an improved contrastive decoding algorithm.
  arXiv  Detail & Related papers  (2024-07-15T15:14:01Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Towards Generating Functionally Correct Code Edits from Natural Language Issue Descriptions [11.327913840111378]
  We introduce Defects4J-NL2Fix, a dataset of 283 Java programs from the popular Defects4J dataset augmented with high-level descriptions of bug fixes. We empirically evaluate the performance of several state-of-the-art LLMs for the this task.
  arXiv  Detail & Related papers  (2023-04-07T18:58:33Z)
• LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
  We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-02-16T18:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.