A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics

• URL: http://arxiv.org/abs/2505.15469v1
• Date: Wed, 21 May 2025 12:45:49 GMT
• Title: A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metrics
• Authors: Jonathan Katzy, Yongcheng Huang, Gopal-Raj Panchu, Maksym Ziemlewski, Paris Loizides, Sander Vermeulen, Arie van Deursen, Maliheh Izadi,
• Abstract summary: We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models.<n>We identify a taxonomy of 26 distinct error categories in model-generated code comments.<n>Our analysis shows that, while these models frequently produce partially correct comments, modern neural fail to reliably differentiate meaningful completions from random noise.
• Score: 10.1653613066577
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Models are essential coding assistants, yet their training is predominantly English-centric. In this study, we evaluate the performance of code language models in non-English contexts, identifying challenges in their adoption and integration into multilingual workflows. We conduct an open-coding study to analyze errors in code comments generated by five state-of-the-art code models, CodeGemma, CodeLlama, CodeQwen1.5, GraniteCode, and StarCoder2 across five natural languages: Chinese, Dutch, English, Greek, and Polish. Our study yields a dataset of 12,500 labeled generations, which we publicly release. We then assess the reliability of standard metrics in capturing comment \textit{correctness} across languages and evaluate their trustworthiness as judgment criteria. Through our open-coding investigation, we identified a taxonomy of 26 distinct error categories in model-generated code comments. They highlight variations in language cohesion, informativeness, and syntax adherence across different natural languages. Our analysis shows that, while these models frequently produce partially correct comments, modern neural metrics fail to reliably differentiate meaningful completions from random noise. Notably, the significant score overlap between expert-rated correct and incorrect comments calls into question the effectiveness of these metrics in assessing generated comments.

Related papers

• IFEvalCode: Controlled Code Generation [69.28317223249358]
  The paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs.<n>The authors present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages.
  arXiv  Detail & Related papers  (2025-07-30T08:08:48Z)
• Fine-Tuning Multilingual Language Models for Code Review: An Empirical Study on Industrial C# Projects [4.3012765978447565]
  This study presents the empirical evaluation of monolingual fine-tuning on the performance of open-source language models (LMs)<n>We fine-tuned three distinct models, CodeReviewer, CodeLlama-7B, and DeepSeek-R1-Distill, on a C# specific dataset combining public benchmarks with industrial repositories.<n>Our results show that monolingual fine-tuning improves model accuracy and relevance compared to multilingual baselines.
  arXiv  Detail & Related papers  (2025-07-25T13:49:24Z)
• Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training [58.696660064190475]
  We find that the existence of code-switching, alternating between different languages within a context, is key to multilingual capabilities.<n>To better explore the power of code-switching for language alignment during pre-training, we investigate the strategy of synthetic code-switching.
  arXiv  Detail & Related papers  (2025-04-02T15:09:58Z)
• On Explaining (Large) Language Models For Code Using Global Code-Based Explanations [45.126233498200534]
  Language Models for Code (LLM4Code) have significantly changed the landscape of software engineering (SE)<n>We introduce code rationales (Code$Q$), a technique with rigorous mathematical underpinning, to identify subsets of tokens that can explain individual code predictions.<n>Our evaluation demonstrates that Code$Q$ is a powerful interpretability method to explain how (less) meaningful input concepts (i.e., natural language particle at') highly impact output generation.
  arXiv  Detail & Related papers  (2025-03-21T01:00:45Z)
• CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models [10.108114365564926]
  State-of-the-art large language models (LLMs) have demonstrated impressive code generation capabilities but struggle with real-world software engineering tasks.<n>Code review comments are often implicit, ambiguous, and colloquial, requiring models to grasp both code and human intent.<n>This challenge calls for evaluating large language models' ability to bridge both technical and conversational contexts.
  arXiv  Detail & Related papers  (2025-03-20T14:07:31Z)
• mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generation [28.531581489405745]
  mHumanEval is an extended benchmark supporting prompts in over 200 natural languages.<n>We provide expert human translations for 15 diverse natural languages (NLs)<n>We conclude by analyzing the multilingual code generation capabilities of state-of-the-art (SOTA) Code LLMs.
  arXiv  Detail & Related papers  (2024-10-19T08:44:26Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.1875460416205]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages.<n>It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total.<n>Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation [30.914387085368734]
  Large Language Models (LLMs) have demonstrated promising capabilities for code generation.<n>In this paper, we aim to investigate the potential linguistic bias through the lens of English and Chinese.
  arXiv  Detail & Related papers  (2024-06-02T03:22:30Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• ICE-Score: Instructing Large Language Models to Evaluate Code [7.556444391696562]
  We propose textttICE-Score, a new evaluation metric via instructing large language models for code assessments. Our metric addresses the limitations of existing approaches by achieving superior correlations with functional correctness and human preferences. Our results demonstrate that our metric surpasses state-of-the-art metrics for code generation.
  arXiv  Detail & Related papers  (2023-04-27T16:38:17Z)
• Code Comment Inconsistency Detection with BERT and Longformer [9.378041196272878]
  Comments, or natural language descriptions of source code, are standard practice among software developers. When the code is modified without an accompanying correction to the comment, an inconsistency between the comment and code can arise. We propose two models to detect such inconsistencies in a natural language inference (NLI) context.
  arXiv  Detail & Related papers  (2022-07-29T02:43:51Z)
• MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages [76.93265104421559]
  We benchmark code generation from natural language commands extending beyond English. We annotated a total of 896 NL-code pairs in three languages: Spanish, Japanese, and Russian. While the difficulties vary across these three languages, all systems lag significantly behind their English counterparts.
  arXiv  Detail & Related papers  (2022-03-16T04:21:50Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.