Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess engines

• URL: http://arxiv.org/abs/2606.13763v1
• Date: Thu, 11 Jun 2026 17:34:00 GMT
• Title: Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess engines
• Abstract summary: Two coding agents were asked to build chess engines without chess knowledge or implementation guidance.<n>We study both through a polyglot case study built around chess engines.<n>We combine per-engine feature analysis, independent Elo assessment, and session trajectories with qualitative analysis of code and transcripts.
• Score: 3.272730424658043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier coding agents now promise end-to-end authorship of complete software systems. Two empirical questions follow: can AI coding-agent teammates program in any target language, including ones with no comparable prior open-source artefact? If so, does language choice still shape the artefact, and along which dimensions? We study both through a polyglot case study built around chess engines: non-trivial multi-component systems that admit a hierarchy of language-agnostic oracles, from exact move-generation correctness to a strength scale (Elo), observable from Rust to Brainfuck. We prompted two frontier agents (Claude Code and Codex) at the capability level, without chess knowledge or implementation guidance, under a documented intervention and stopping policy. The agents produced 34 chess engines spanning 17 primary programming languages, from mainstream to specialised, domain-specific, legacy, and esoteric targets. We combine per-engine feature analysis, independent Elo assessment, and session trajectories with qualitative analysis of code and transcripts. Frontier coding agents are genuinely polyglot: every language we tried produced at least one feature-rich working engine, several with no prior open-source counterpart of comparable scope (e.g., LaTeX), and the code is synthesised from scratch rather than copied. Yet language choice still matters: strong playing strength is only reachable in mainstream compiled languages, cost and engineering effort grow sharply as the language becomes more exotic, and feature choices shift across language families. Agents validate their own work unprompted, but their strength self-estimates are biased and a few engines cheated by calling a chess library. Programming language is no longer about whether AI teammates can build a working system, but about performance, cost, what gets built, and how much human supervision validation still needs.

Related papers

• The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages [13.509521852529728]
  We evaluate five recent models on programming problems in Python, Java, Rust, and OCaml.<n>We show that there can be stark variation in token consumption that is consistent across models.
  arXiv  Detail & Related papers  (2026-07-24T16:41:59Z)
• Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality [45.28284966022665]
  We study how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude.<n>We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian.<n>We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, and issues detected by static analysis tools.
  arXiv  Detail & Related papers  (2026-07-16T10:35:25Z)
• Large Language Models for Multilingual Code Intelligence: A Survey [15.859504186659825]
  This survey focuses on two key tasks: multilingual code generation from shared natural-language requirements, and multilingual code translation that preserves semantics across languages.<n>It reviews representative methods, benchmarks, and evaluation metrics, and highlights challenges and opportunities for trustworthy cross-language generalization.
  arXiv  Detail & Related papers  (2026-04-27T20:20:26Z)
• Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders [51.380449540006985]
  Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear.<n>Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language?<n>We analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs.
  arXiv  Detail & Related papers  (2025-11-13T22:51:06Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.1875460416205]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages.<n>It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total.<n>Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• Symbolic Learning Enables Self-Evolving Agents [55.625275970720374]
  We introduce agent symbolic learning, a systematic framework that enables language agents to optimize themselves on their own. Agent symbolic learning is designed to optimize the symbolic network within language agents by mimicking two fundamental algorithms in connectionist learning. We conduct proof-of-concept experiments on both standard benchmarks and complex real-world tasks.
  arXiv  Detail & Related papers  (2024-06-26T17:59:18Z)
• CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation [58.84212778960507]
  CodeGRAG builds the graphical view of code blocks based on the control flow and data flow of them to better interpret the programming domain knowledge.<n>CodeGRAG significantly improves the code generation ability of LLMs and can even offer performance gain for cross-lingual code generation.
  arXiv  Detail & Related papers  (2024-05-03T02:48:55Z)
• Agents: An Open-source Framework for Autonomous Language Agents [98.91085725608917]
  We consider language agents as a promising direction towards artificial general intelligence. We release Agents, an open-source library with the goal of opening up these advances to a wider non-specialist audience.
  arXiv  Detail & Related papers  (2023-09-14T17:18:25Z)
• From Copilot to Pilot: Towards AI Supported Software Development [3.0585424861188844]
  We study the limitations of AI-supported code completion tools like Copilot and offer a taxonomy for understanding the classification of AI-supported code completion tools in this space. We then conduct additional investigation to determine the current boundaries of AI-supported code completion tools like Copilot. We conclude by providing a discussion on challenges for future development of AI-supported code completion tools to reach the design level of abstraction in our taxonomy.
  arXiv  Detail & Related papers  (2023-03-07T18:56:52Z)
• xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval [32.60391966381949]
  We introduce xCodeEval, the largest executable multilingual multitask benchmark to date. It features a total of $7$ tasks involving code understanding, generation, translation and retrieval. xCodeEval adopts an execution-based evaluation and offers a multilingual code execution engine, ExecEval.
  arXiv  Detail & Related papers  (2023-03-06T10:08:51Z)
• A Systematic Evaluation of Large Language Models of Code [88.34057460577957]
  Large language models (LMs) of code have recently shown tremendous promise in completing code and synthesizing code from natural language descriptions. The current state-of-the-art code LMs are not publicly available, leaving many questions about their model and data design decisions. Although Codex is not open-source, we find that existing open-source models do achieve close results in some programming languages. We release a new model, PolyCoder, with 2.7B parameters based on the GPT-2 architecture, which was trained on 249GB of code across 12 programming languages on a single machine.
  arXiv  Detail & Related papers  (2022-02-26T15:53:55Z)
• AVATAR: A Parallel Corpus for Java-Python Program Translation [77.86173793901139]
  Program translation refers to migrating source code from one language to another. We present AVATAR, a collection of 9,515 programming problems and their solutions written in two popular languages, Java and Python.
  arXiv  Detail & Related papers  (2021-08-26T05:44:20Z)
• Unsupervised Translation of Programming Languages [19.56070393390029]
  A transcompiler, also known as source-to-source, is a system that converts source code from a high-level programming language to another. We train our model on source code from open source GitHub projects, and show that it can translate functions between C++, Java, and Python with high accuracy.
  arXiv  Detail & Related papers  (2020-06-05T15:28:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.