Beyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation

• URL: http://arxiv.org/abs/2605.02195v3
• Date: Fri, 08 May 2026 04:53:18 GMT
• Title: Beyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation
• Abstract summary: Large Language Models (LLMs) have achieved remarkable success in automated code translation.<n>In this paper, we demonstrate that a significant number of reported failures in code translation are not due to incorrect logic, but rather evaluation-induced errors.
• Score: 2.3411689215528346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have achieved remarkable success in automated code translation. While prior work has focused on improving translation accuracy through advanced prompting and iterative repair, the reliability of the underlying evaluation frameworks has received less attention. In this paper, we demonstrate that a significant number of reported failures in code translation are not due to incorrect logic, but rather evaluation-induced errors stemming from improper compilation flags, missing library links, and unconfigured runtime environments. We conduct a large-scale empirical study across five programming languages (C, C++, Java, Python, Go) and three benchmarks (Avatar, CodeNet, EvalPlus), covering 6,164 translations generated by GPT-4o, DeepSeek-Coder, and Magicoder. Our analysis identifies and categorizes common false negatives, distinguishing pipeline-induced failures that affect any model from model-dependent behaviors that vary across LLMs. Our findings highlight the necessity for transparent, configuration-aware evaluation standards to accurately assess progress in LLM-based code translation.

Related papers

• Holistic Evaluation of State-of-the-Art LLMs for Code Generation [5.504955093712013]
  DeepSeek-R1 and GPT-4.1 consistently outperform others in terms of correctness, efficiency, and robustness.<n>We identify common failure scenarios such as syntax errors, logical flaws, and suboptimal algorithms.
  arXiv  Detail & Related papers  (2025-12-19T23:29:05Z)
• CodeFlowLM: Incremental Just-In-Time Defect Prediction with Pretrained Language Models and Exploratory Insights into Defect Localization [1.3878785822379076]
  This work introduces CodeFlowLM, an incremental learning framework for Just-In-Time Software Defect Prediction (JIT-SDP)<n>Unlike traditional online learners, CodeFlowLM employs continual fine-tuning to address concept drift, class imbalance, and verification latency without retraining from scratch.<n>We extend the analysis to benchmarking Large Language Models (LLMs) such as GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Pro against attention-based models.
  arXiv  Detail & Related papers  (2025-11-28T22:18:46Z)
• LLM-GUARD: Large Language Model-Based Detection and Repair of Bugs and Security Vulnerabilities in C++ and Python [0.0]
  Large Language Models (LLMs) such as ChatGPT-4, Claude 3, and LLaMA 4 are increasingly embedded in software/application development.<n>This study presents a systematic, empirical evaluation of these three leading LLMs using a benchmark of programming errors, classic security flaws, and advanced, production-grade bugs in C++ and Python.
  arXiv  Detail & Related papers  (2025-08-22T14:30:24Z)
• Function-to-Style Guidance of LLMs for Code Translation [59.487054943812836]
  We propose F2STrans, a function-to-style guiding paradigm designed to improve the performance of large language models in code translation.<n>Our approach comprises two key stages: (1) Functional learning, which optimize translation correctness using high-quality source-target code pairs.<n>We introduce a novel code translation benchmark that includes up-to-date source code, extensive test cases, and manually annotated ground-truth translations.
  arXiv  Detail & Related papers  (2025-07-15T08:25:02Z)
• NL in the Middle: Code Translation with LLMs and Intermediate Representations [56.77064674776534]
  Large language models (LLMs) produce buggy code translations.<n>One promising avenue to improve translation accuracy is through intermediate representations.<n>We investigate whether LLM-based code translation can benefit from intermediate representations.
  arXiv  Detail & Related papers  (2025-07-11T14:29:21Z)
• D-LiFT: Improving LLM-based Decompiler Backend via Code Quality-driven Fine-tuning [49.16469288280772]
  Decompilers reconstruct human-readable source code from binaries.<n>Despite recent advances, their outputs often suffer from syntactic and semantic errors and remain difficult to read.<n>With the advent of large language models (LLMs), researchers began to explore the potential of LLMs to refine decompiler output.<n>We present D-LIFT, an enhanced decompiler-LLM pipeline with fine-tuned reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-11T19:09:08Z)
• Lost in Literalism: How Supervised Training Shapes Translationese in LLMs [51.04435855143767]
  Large language models (LLMs) have achieved remarkable success in machine translation.<n>However, translationese, characterized by overly literal and unnatural translations, remains a persistent challenge.<n>We introduce methods to mitigate these biases, including polishing golden references and filtering unnatural training instances.
  arXiv  Detail & Related papers  (2025-03-06T12:14:45Z)
• RustRepoTrans: Repository-level Code Translation Benchmark Targeting Rust [50.65321080814249]
  RustRepoTrans is the first repository-level context code translation benchmark targeting incremental translation.<n>We evaluate seven representative LLMs, analyzing their errors to assess limitations in complex translation scenarios.
  arXiv  Detail & Related papers  (2024-11-21T10:00:52Z)
• Semantic Alignment-Enhanced Code Translation via an LLM-Based Multi-Agent System [24.52067108242477]
  Code translation is crucial for software migration, system ablation, and cross-platform development.<n>Traditional rule-based methods rely on manually-written rules, which can be time-consuming and often result in less readable code.<n>More recently, the advance of Large Language Models (LLMs) further boosts learning-based code translation.<n>We propose a novel multi-agent system TRANSAGENT, which enhances LLM-based code translation by fixing the syntax errors and semantic errors.
  arXiv  Detail & Related papers  (2024-09-30T02:53:03Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Towards Translating Real-World Code with LLMs: A Study of Translating to Rust [13.743967357458287]
  Large language models (LLMs) show promise in code translation due to their ability to write code in most programming languages.<n>We conduct our study on code extracted from real-world open source projects.<n> FLOURINE is an end-to-end code translation tool that uses differential fuzzing to check if a Rust translation is I/O equivalent to the original source program.
  arXiv  Detail & Related papers  (2024-05-19T10:54:03Z)
• Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning [57.323716555996114]
  Off-target translation remains an unsolved problem, especially for low-resource languages. Recent works have either designed advanced prompting strategies to highlight the functionality of translation instructions or exploited the in-context learning ability of LLMs. In this work, we design a two-stage fine-tuning algorithm to improve the instruction-following ability (especially the translation direction) of LLMs.
  arXiv  Detail & Related papers  (2024-03-21T13:47:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.