Kaizen: Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications

• URL: http://arxiv.org/abs/2607.04058v1
• Date: Sat, 04 Jul 2026 23:52:39 GMT
• Title: Kaizen: Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications
• Abstract summary: We present a framework for evaluating correctness of large language models (LLMs) translated to OpenMP, OpenACC, Kokkos or SYCL.<n>We evaluate on-to-OpenMP translation of 16 scientific applications using three fine-tuned LLMs at kernel-level and full-program granularity.
• Score: 2.4948485991003158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly used to port scientific codes across heterogeneous high-performance computing (HPC) programming models, such as translating CUDA to OpenMP, OpenACC, Kokkos or SYCL. However, current evaluations use compilation success, token-level similarity, or developer-written tests from static benchmarks, which cannot reliably ensure behavioral correctness. We present Kaizen, a metamorphic fuzzing and differential testing framework for evaluating the correctness of LLM-translated HPC code. Kaizen uses metamorphic fuzzing via source-code mutation to generate semantically equivalent programs, grammar-based input fuzzing to explore behavioral diversity, and differential testing to expose semantic divergences between original and translated applications that compile and pass developer-written tests yet produce incorrect scientific results. We evaluate Kaizen on CUDA-to-OpenMP translation of 16 scientific applications from seven domains using three fine-tuned LLMs at kernel-level and full-program granularity. Our evaluation reveals that (1) compilation success is a poor proxy for correctness; (2) LLM-translated programs exhibit systematic compile-time error patterns, with nine categories for kernel-level translation and 27 for full-program translation; (3) semantic errors that survive compilation are often input-dependent and require differential testing to expose; and (4) full-program translation is substantially harder than kernel-level translation. These findings highlight the need for correctness-oriented evaluation of LLM-assisted HPC code translations.

Related papers

• Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation [21.642589721687724]
  runtime efficiency has become increasingly important for program quality, alongside functional correctness.<n>Our work proposes SwiftTrans, a code translation framework comprising two key stages: Multi-Perspective Exploration and Difference-Aware Selection.<n> Experimental results across all three benchmarks show that SwiftTrans achieves consistent improvements in both correctness and runtime efficiency.
  arXiv  Detail & Related papers  (2026-06-16T08:49:56Z)
• ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation [5.103450268268085]
  We present ParBench, a kernel-centric benchmark framework for evaluating parallel API translation under executable conditions.<n>ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only kernels.
  arXiv  Detail & Related papers  (2026-06-09T19:00:09Z)
• From I/O to Code with Discovery Agent [103.88427301265669]
  We propose DIO-Agent, a discovery agent for IO2Code.<n>Our method frames IO2Code as an evolutionary search over discrete program space.<n>Extensive experiments show that DIO-Agent consistently outperforms both traditional program-by-example method and SOTA evolution-agent baselines.
  arXiv  Detail & Related papers  (2026-05-14T18:57:32Z)
• Beyond BLEU: A Semantic Evaluation Method for Code Translation [2.3802148866231057]
  We propose a novel evaluation methodology for code translation tasks, emphasizing semantic equivalence over surface-level string similarity.<n>We introduce a semantic correctness score, defined as the proportion of translations that produce correct execution outcomes.<n>Our findings show that LLM-based approaches significantly outperform ones, while BLEU scores show negligible correlation with semantic correctness.
  arXiv  Detail & Related papers  (2026-05-06T17:14:33Z)
• Beyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation [2.3411689215528346]
  Large Language Models (LLMs) have achieved remarkable success in automated code translation.<n>In this paper, we demonstrate that a significant number of reported failures in code translation are not due to incorrect logic, but rather evaluation-induced errors.
  arXiv  Detail & Related papers  (2026-05-04T03:49:58Z)
• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• Function-to-Style Guidance of LLMs for Code Translation [59.487054943812836]
  We propose F2STrans, a function-to-style guiding paradigm designed to improve the performance of large language models in code translation.<n>Our approach comprises two key stages: (1) Functional learning, which optimize translation correctness using high-quality source-target code pairs.<n>We introduce a novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth translations.
  arXiv  Detail & Related papers  (2025-07-15T08:25:02Z)
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning [49.16469288280772]
  Decompilers reconstruct human-readable source code from binaries.<n>Despite recent advances, their outputs often suffer from syntactic and semantic errors and remain difficult to read.<n>With the advent of large language models (LLMs), researchers began to explore the potential of LLMs to refine decompiler output.<n>We present D-LIFT, an enhanced decompiler-LLM pipeline with fine-tuned reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T19:09:08Z)
• LLM-Assisted Translation of Legacy FORTRAN Codes to C++: A Cross-Platform Study [38.73914653312889]
  Large Language Models (LLMs) are increasingly being leveraged for generating and translating scientific computer codes.<n>Here, we studied the applicability of LLM-based translation of Fortran to C++ as a step towards building an agentic-workflow.<n>We statistically quantified the compilation accuracy of the translated C++ codes, measured the similarity of the LLM translated code to the human translated C++ code, and statistically quantified the output similarity of the Fortran to C++ translation.
  arXiv  Detail & Related papers  (2025-04-21T20:34:37Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• Guess & Sketch: Language Model Guided Transpilation [59.02147255276078]
  Learned transpilation offers an alternative to manual re-writing and engineering efforts. Probabilistic neural language models (LMs) produce plausible outputs for every input, but do so at the cost of guaranteed correctness. Guess & Sketch extracts alignment and confidence information from features of the LM then passes it to a symbolic solver to resolve semantic equivalence.
  arXiv  Detail & Related papers  (2023-09-25T15:42:18Z)
• LeTI: Learning to Generate from Textual Interactions [60.425769582343506]
  We explore LMs' potential to learn from textual interactions (LETI) that not only check their correctness with binary labels but also pinpoint and explain errors in their outputs through textual feedback. Our focus is the code generation task, where the model produces code based on natural language instructions. LETI iteratively fine-tunes the model, using the objective LM, on a concatenation of natural language instructions, LM-generated programs, and textual feedback.
  arXiv  Detail & Related papers  (2023-05-17T15:53:31Z)
• LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
  We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-02-16T18:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.