How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks

• URL: http://arxiv.org/abs/2511.04115v1
• Date: Thu, 06 Nov 2025 07:06:20 GMT
• Title: How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks
• Authors: Ruksit Rojpaisarnkit, Youmei Fan, Kenichi Matsumoto, Raula Gaikovina Kula,
• Abstract summary: This paper investigates whether the English language proficiency itself affects the proficiency and correctness of code generated by Large Language Models (LLMs)<n>We varied the English proficiency of prompts from basic to advanced for 164 programming tasks and measured the resulting code proficiency and correctness.<n>While the effect on the resulting code proficiency was model-dependent, we found that higher-proficiency prompts consistently yielded more correct code across all models.
• Score: 3.487093088832285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the widespread adoption of Foundation Model (FM)-powered tools in software engineering, the natural language prompt has become a critical interface between developers and Large Language Models (LLMs). While much research has focused on prompt structure, the natural language proficiency is an underexplored factor that can influence the quality of generated code. This paper investigates whether the English language proficiency itself independent of the prompting technique affects the proficiency and correctness of code generated by LLMs. Using the HumanEval dataset, we systematically varied the English proficiency of prompts from basic to advanced for 164 programming tasks and measured the resulting code proficiency and correctness. Our findings show that LLMs default to an intermediate (B2) natural language level. While the effect on the resulting code proficiency was model-dependent, we found that higher-proficiency prompts consistently yielded more correct code across all models. These results demonstrate that natural language proficiency is a key lever for controlling code generation, helping developers tailor AI output and improve the reliability of solutions.

Related papers

• CodeSimpleQA: Scaling Factuality in Code Large Language Models [55.705748501461294]
  We present CodeSimpleQA, a comprehensive benchmark designed to evaluate the factual accuracy of code LLMs in answering code-related questions.<n>We also create CodeSimpleQA-Instruct, a large-scale instruction corpus with 66M samples, and develop a post-training framework combining supervised fine-tuning and reinforcement learning.
  arXiv  Detail & Related papers  (2025-12-22T14:27:17Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• Bridging LLM-Generated Code and Requirements: Reverse Generation technique and SBC Metric for Developer Insights [0.0]
  This paper introduces a novel scoring mechanism called the SBC score.<n>It is based on a reverse generation technique that leverages the natural language generation capabilities of Large Language Models.<n>Unlike direct code analysis, our approach reconstructs system requirements from AI-generated code and compares them with the original specifications.
  arXiv  Detail & Related papers  (2025-02-11T01:12:11Z)
• Large Language Models are Interpretable Learners [53.56735770834617]
  In this paper, we show a combination of Large Language Models (LLMs) and symbolic programs can bridge the gap between expressiveness and interpretability. The pretrained LLM with natural language prompts provides a massive set of interpretable modules that can transform raw input into natural language concepts. As the knowledge learned by LSP is a combination of natural language descriptions and symbolic rules, it is easily transferable to humans (interpretable) and other LLMs.
  arXiv  Detail & Related papers  (2024-06-25T02:18:15Z)
• Synthetic Programming Elicitation for Text-to-Code in Very Low-Resource Programming and Formal Languages [21.18996339478024]
  We introduce emphsynthetic programming elicitation and compilation (SPEAC) SPEAC produces syntactically correct programs more frequently and without sacrificing semantic correctness. We empirically evaluate the performance of SPEAC in a case study for the UCLID5 formal verification language.
  arXiv  Detail & Related papers  (2024-06-05T22:16:19Z)
• From Effectiveness to Efficiency: Uncovering Linguistic Bias in Large Language Model-based Code Generation [30.914387085368734]
  Large Language Models (LLMs) have demonstrated promising capabilities for code generation.<n>In this paper, we aim to investigate the potential linguistic bias through the lens of English and Chinese.
  arXiv  Detail & Related papers  (2024-06-02T03:22:30Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to better interpret the programming domain knowledge.<n>CodeGRAG significantly improves the code generation ability of LLMs and can even offer performance gain for cross-lingual code generation.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Cross-lingual Transfer in Programming Languages: An Extensive Empirical Study [5.350495525141013]
  Large language models (LLMs) have achieved state-of-the-art performance in various software engineering tasks.<n>critical languages, such as Rust and Swift, remain low-resource due to limited openly available code.<n>We develop a performance prediction model to guess the best source languages for a given target and task.
  arXiv  Detail & Related papers  (2023-10-25T19:04:33Z)
• Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach [12.214585409361126]
  Large language models (LLMs)- based code generation is a complex and powerful black-box model. We propose a novel causal graph-based representation of the prompt and the generated code. We illustrate the insights that our framework can provide by studying over 3 popular LLMs with over 12 prompt adjustment strategies.
  arXiv  Detail & Related papers  (2023-10-10T14:56:26Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.