ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation

• URL: http://arxiv.org/abs/2607.22588v1
• Date: Tue, 09 Jun 2026 19:00:09 GMT
• Title: ParBench: A Benchmark for Reliable Evaluation of LLM Parallel Code Translation
• Abstract summary: We present ParBench, a kernel-centric benchmark framework for evaluating parallel API translation under executable conditions.<n>ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only kernels.
• Score: 5.103450268268085
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern compute-intensive software must migrate across a changing ecosystem of accelerators, programming APIs, compiler stacks, and portability layers, including CUDA, OpenMP, OpenCL, and OpenMP target offload. Large language models and autonomous coding agents are increasingly proposed for such migration, but the field lacks reliable ways to measure whether they preserve the low-level parallel semantics that make translations behaviorally valid, including thread indexing, synchronization, memory management, host-device coordination, and API-specific execution structure. We present ParBench, a kernel-centric benchmark framework for evaluating LLM-based parallel API translation under executable, reproducible conditions. ParBench fixes the surrounding build, run, and verification infrastructure through declarative benchmark specifications and asks models to translate only the computational kernels. It draws on multiple open-source HPC suites and covers representative cross-API translation directions among CUDA, OpenMP, OpenCL, and OpenMP target offload. To test whether success reflects robust translation rather than surface-form memorization, ParBench includes AST-driven, intended behavior-preserving, baseline-validated source augmentation. Evaluations on state-of-the-art open and proprietary LLMs show persistent barriers to reliable parallel code translation, including direction asymmetry, multi-file coordination, incomplete API adaptation, and uneven robustness to source-level perturbations. Code is available at https://github.com/Scientific-Computing-Lab/ParBench.

Related papers

• ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs [48.87461975674182]
  We introduce the Parallel Vision-Language (ParVL) scaling framework for Multimodal Large Language Models.<n>ParVL scales parallel computation by reusing the existing ViT and LLM backbone parameters across multiple vision and language branches.<n>We show that ParVL improves overall multimodal performance over same-recipe single-branch baselines.
  arXiv  Detail & Related papers  (2026-08-04T17:59:58Z)
• CodegenBench: Can LLMs Write Efficient Code Across Architectures? [17.06172900876354]
  Large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments.<n>CodegenBench is a benchmark suite designed to evaluate the generation of efficient parallel code across three distinct hardware platforms.
  arXiv  Detail & Related papers  (2026-06-01T12:55:10Z)
• Environment-Aware Code Generation: How far are We? [52.69113158357018]
  It is unclear whether large language models (LLMs) can reliably generate executable code tailored to a user's specific environment.<n>We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations.<n>Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability.
  arXiv  Detail & Related papers  (2026-01-18T04:58:15Z)
• OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms [13.343925256921722]
  We introduce OMPILOT, a domain-specific encoder-decoder transformer tailored for translating C++ code into OpenMP.<n>OMPBLEU is a novel composite metric crafted to assess the correctness and quality of OpenMP parallel constructs.
  arXiv  Detail & Related papers  (2025-11-05T21:21:15Z)
• HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration [13.53425131505526]
  Deep learning has driven exponential increases in model parameters and computational demands.<n> NVIDIA GPUs and their-based software ecosystem provide robust support for parallel computing.<n>The ecosystem has established a dominant position in the field of parallel software.<n> translating code to other platforms poses significant challenges due to differences parallel programming paradigms and hardware.
  arXiv  Detail & Related papers  (2025-06-12T06:48:33Z)
• ExeCoder: Empowering Large Language Models with Executability Representation for Code Translation [57.604506522287814]
  Existing large language models (LLMs) only learn the contextual semantics of code during pre-training.<n>We propose ExeCoder to utilize executability representations such as functional semantics, syntax structures, and variable dependencies.<n>We show that ExeCoder achieves state-of-the-art performance in code translation, surpassing existing open-source code LLMs by over 10.88% to 38.78% and over 27.44% to 42.97% on two metrics.
  arXiv  Detail & Related papers  (2025-01-30T16:18:52Z)
• LASSI: An LLM-based Automated Self-Correcting Pipeline for Translating Parallel Scientific Codes [0.23301643766310373]
  LASSI is designed to translate between parallel programming languages by bootstrapping existing closed- or open-source LLMs.<n>LASSI incorporates autonomous enhancement through self-correcting loops where errors encountered during the compilation and execution of generated code are fed back to the LLM.
  arXiv  Detail & Related papers  (2024-06-30T19:36:04Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• L2MAC: Large Language Model Automatic Computer for Extensive Code Generation [52.81694565226513]
  Transformer-based large language models (LLMs) are constrained by the fixed context window of the underlying transformer architecture.<n>This paper presents L2MAC, the first practical LLM-based general-purpose stored-program automatic computer (von Neumann architecture) framework, for long and consistent output generation.
  arXiv  Detail & Related papers  (2023-10-02T16:55:19Z)
• Advising OpenMP Parallelization via a Graph-Based Approach with Transformers [2.393682571484038]
  We propose a novel approach, called OMPify, to detect and predict the OpenMP pragmas and shared-memory attributes in parallel code. OMPify is based on a Transformer-based model that leverages a graph-based representation of source code. Our results demonstrate that OMPify outperforms existing approaches, the general-purposed and popular ChatGPT and targeted PragFormer models.
  arXiv  Detail & Related papers  (2023-05-16T16:56:10Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)
• QParallel: Explicit Parallelism for Programming Quantum Computers [62.10004571940546]
  We present a language extension for parallel quantum programming. QParallel removes ambiguities concerning parallelism in current quantum programming languages. We introduce a tool that guides programmers in the placement of parallel regions by identifying the subroutines that profit most from parallelization.
  arXiv  Detail & Related papers  (2022-10-07T16:35:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.