TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation

• URL: http://arxiv.org/abs/2603.16479v2
• Date: Thu, 19 Mar 2026 03:05:38 GMT
• Title: TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation
• Abstract summary: We present textbftextsctrace, the first benchmark to explicitly assess efficiency in Large Language Models (LLMs)<n>textsctrace includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency degradations often overlooked by small-scale tests.<n>Our results establish efficiency as an essential dimension of code translation and position textsctrace as a principled foundation for efficiency-oriented evaluation.
• Score: 15.051178553018746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Large Language Models (LLMs) have substantially improved the functional correctness of code translation, the critical dimension of \textit{execution efficiency} remains overlooked. We present \textbf{\textsc{trace}}, the first benchmark to explicitly assess efficiency in LLM-translated code. \textsc{trace} includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency degradations often overlooked by small-scale tests. Using \textsc{trace}, we conduct an extensive evaluation of 28 representative LLMs and highlight several key insights: 1) Correctness is not a reliable proxy for efficiency: the correctness leader \textit{Claude-4-think} achieves only mid-level time efficiency, outperformed by smaller open-source LLMs such as \textit{Qwen2.5-Coder-14B-Instruct}. 2) Inefficiency is both prevalent and patterned: 23.5\% of correct translations exhibit pronounced inefficiency, distributed across algorithmic faults (11.9\%), language construct mismatches (66.4\%), and resource mismanagement (21.7\%). 3) Inference-time prompt strategies bring only modest improvements, suggesting that current LLMs lack intrinsic efficiency awareness. Together, our results establish efficiency as an essential dimension of code translation and position \textsc{trace} as a principled foundation for efficiency-oriented evaluation.

Related papers

• Chiseling Out Efficiency: Structured Skeleton Supervision for Efficient Code Generation [61.72280804975896]
  Large Language Models (LLMs) are capable of generating syntactically correct and functionally complete programs, greatly streamlining software development.<n>Existing approaches to bridging this efficiency gap typically involve either iteratively optimizing code after generation or fine-tuning models on corpora of efficient code.<n>EffiSkel is an efficiency skeleton-guided framework that explicitly extracts and learns efficiency skeletons-abstract, reusable structural patterns efficient code.
  arXiv  Detail & Related papers  (2026-06-05T01:49:20Z)
• PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization [43.72919370258115]
  We introduce PerfCodeBench, an executable benchmark for evaluating large language models (LLMs) on high-performance code optimization.<n>The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenecks.<n>Our evaluation shows a clear gap between model-generated code and expert-optimized implementations.
  arXiv  Detail & Related papers  (2026-05-13T08:10:26Z)
• Prism: Efficient Test-Time Scaling via Hierarchical Search and Self-Verification for Discrete Diffusion Language Models [96.0074341403456]
  Inference-time compute has re-emerged as a practical way to improve LLM reasoning.<n>Most test-time scaling (TTS) algorithms rely on autoregressive decoding.<n>We propose Prism, an efficient TTS framework for dLLMs.
  arXiv  Detail & Related papers  (2026-02-02T09:14:51Z)
• DEPO: Dual-Efficiency Preference Optimization for LLM Agents [75.6723341304463]
  We propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps.<n>Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance.
  arXiv  Detail & Related papers  (2025-11-19T12:38:43Z)
• Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs [57.10533368622962]
  Self-correction of large language models (LLMs) emerges as a critical component for enhancing their reasoning performance.<n>This study introduces CorrectBench, a benchmark developed to evaluate the effectiveness of self-correction strategies.<n>Our findings reveal that: 1) Self-correction methods can improve accuracy, especially for complex reasoning tasks; 2) Mixing different self-correction strategies yields further improvements, though it reduces efficiency; and 3) Reasoning LLMs (e.g., DeepSeek-R1) have limited optimization under additional self-correction methods and have high time costs.
  arXiv  Detail & Related papers  (2025-10-17T02:40:19Z)
• TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation [15.302454413096335]
  We introduce TRACY, the first comprehensive benchmark designed to evaluate the execution efficiency of LLM-translated code.<n>The resulting benchmark comprises 1,011 code translation tasks across C++, Java, and Python.<n>Our work underscores the necessity of jointly optimizing for correctness and efficiency in future LLM-based code translation.
  arXiv  Detail & Related papers  (2025-08-15T13:33:52Z)
• On the Effectiveness of LLM-as-a-judge for Code Generation and Summarization [54.965787768076254]
  Large Language Models have been recently exploited as judges for complex natural language processing tasks, such as Q&A.<n>We study the effectiveness of LLMs-as-a-judge for two code-related tasks, namely code generation and code summarization.
  arXiv  Detail & Related papers  (2025-07-22T13:40:26Z)
• Function-to-Style Guidance of LLMs for Code Translation [59.487054943812836]
  We propose F2STrans, a function-to-style guiding paradigm designed to improve the performance of large language models in code translation.<n>Our approach comprises two key stages: (1) Functional learning, which optimize translation correctness using high-quality source-target code pairs.<n>We introduce a novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth translations.
  arXiv  Detail & Related papers  (2025-07-15T08:25:02Z)
• BenTo: Benchmark Task Reduction with In-Context Transferability [32.561978389905434]
  This paper investigates how to efficiently reduce the tasks used to benchmark large language models (LLMs) We propose a practically efficient metric for estimating the transferability between two tasks via in-context learning (ICL)
  arXiv  Detail & Related papers  (2024-10-17T17:41:15Z)
• Bridging LLMs and KGs without Fine-Tuning: Intermediate Probing Meets Subgraph-Aware Entity Descriptions [49.36683223327633]
  Large Language Models (LLMs) encapsulate extensive world knowledge and exhibit powerful context modeling capabilities.<n>We propose a novel framework that synergizes the strengths of LLMs with robust knowledge representation to enable effective and efficient KGC.<n>We achieve a 47% relative improvement over previous methods based on non-fine-tuned LLMs and, to our knowledge, are the first to achieve classification performance comparable to fine-tuned LLMs.
  arXiv  Detail & Related papers  (2024-08-13T10:15:55Z)
• ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? [12.862825053595934]
  ECCO is a benchmark for evaluating program efficiency via two paradigms: natural language (NL) based code generation and history-based code editing. We find that adding execution information often helps maintain functional correctness, and NL feedback enhances more on efficiency.
  arXiv  Detail & Related papers  (2024-07-19T05:47:40Z)
• How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark [39.13045037676502]
  Development of large language models (LLMs) has significantly pushed the frontiers of program synthesis.<n>Most evaluation frameworks focus on the (functional) correctness of generated code; efficiency, as an important measure of code quality, has been overlooked in existing evaluations.<n>We develop ENAMEL, a rigorous and high-standard benchmark for evaluating the capability of LLMs in generating efficient code.
  arXiv  Detail & Related papers  (2024-06-10T04:19:20Z)
• Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy [48.29181662640212]
  Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. We consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs.
  arXiv  Detail & Related papers  (2024-02-20T08:41:23Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.