Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language

• URL: http://arxiv.org/abs/2605.15607v1
• Date: Fri, 15 May 2026 04:37:31 GMT
• Title: Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language
• Abstract summary: Python outperforms PyLang by up to 19% across all configurations.<n>Models possess language-agnostic understanding but cannot express it in an unfamiliar language.
• Score: 9.529147118376464
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) achieve high pass rates on code generation benchmarks, yet whether they can transfer this ability to languages absent from pretraining remains poorly understood. We introduce PyLang, a minimal imperative language absent from all pretraining corpora, and evaluate frontier models zero-shot and fine-tuned Qwen3 (4B, 8B, 32B) on 352 problems. We find that fine-tuning quickly teaches syntax but fails to transfer semantic competence: Python outperforms PyLang by up to 19% across all configurations, and no intervention (multi-task learning, preference tuning, code infilling, or latent-space objectives) closes the gap. An LLM judge reveals that frontier models select an identical algorithm to Python 80% of the time, yet cannot translate it into a working PyLang implementation., and CKA analysis confirms that fine-tuned models converge to nearly identical internal representations across languages (CKA > 0.97) while diverging at the output stage. We term this the implementation fidelity gap: models possess language-agnostic algorithmic understanding but cannot express it in an unfamiliar language. Our findings highlight the need for training methods that decouple reasoning from language-specific realization.

Related papers

• Learn2Zinc: Fine-tuning Small Language Models for Text-to-Model Translation in MiniZinc [3.9124823111588163]
  We investigate whether targeted fine-tuning can teach small language models to generate correct MiniZinc models from natural language problem descriptions.<n>Key finding is that syntax errors dominate failures when working with this domain specific language.<n>We propose a cross-model error bootstrapping approach that collects syntax errors from multiple LLM runs and leverage those to curate an error correction training dataset.
  arXiv  Detail & Related papers  (2026-05-14T22:55:14Z)
• Language Steering for Multilingual In-Context Learning [10.932074928744568]
  Large language models' performance on non-English languages remains substantially inferior to English.<n>We propose language vectors -- a training-free language steering approach.<n>We show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested.
  arXiv  Detail & Related papers  (2026-02-02T16:52:09Z)
• LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them? [3.809788214307542]
  We identify and characterize two key failure modes: the multilingual transfer bottleneck and the language consistency bottleneck.<n>We extend logit lens analysis to track language probabilities layer by layer and compute cross-lingual semantic similarity of hidden states.<n>This is the first approach to leverage layer-localization of language control for efficient multilingual adaptation.
  arXiv  Detail & Related papers  (2026-01-27T19:38:12Z)
• Anka: A Domain-Specific Language for Reliable LLM Code Generation [0.0]
  Large Language Models (LLMs) exhibit systematic errors on complex, multi-step programming tasks.<n>We introduce Anka, a domain-specific language () for data transformation pipelines designed with explicit, constrained syntax.<n>Anka achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems.
  arXiv  Detail & Related papers  (2025-12-29T05:28:17Z)
• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.1875460416205]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages.<n>It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total.<n>Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora.<n>But can these models relate corresponding concepts across languages, i.e., be crosslingual?<n>This study evaluates state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed? [40.13166574854085]
  We investigate the minimal amount of multilinguality required to elicit cross-lingual generalisation in English-centric large language models. We find that multilingual instruction tuning with as few as two to three languages is both necessary and sufficient to elicit effective cross-lingual generalisation.
  arXiv  Detail & Related papers  (2023-12-20T00:49:52Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Detecting Languages Unintelligible to Multilingual Models through Local Structure Probes [15.870989191524094]
  We develop a general approach that requires only unlabelled text to detect which languages are not well understood by a cross-lingual model. Our approach is derived from the hypothesis that if a model's understanding is insensitive to perturbations to text in a language, it is likely to have a limited understanding of that language.
  arXiv  Detail & Related papers  (2022-11-09T16:45:16Z)
• Lifting the Curse of Multilinguality by Pre-training Modular Transformers [72.46919537293068]
  multilingual pre-trained models suffer from the curse of multilinguality, which causes per-language performance to drop as they cover more languages. We introduce language-specific modules, which allows us to grow the total capacity of the model, while keeping the total number of trainable parameters per language constant. Our approach enables adding languages post-hoc with no measurable drop in performance, no longer limiting the model usage to the set of pre-trained languages.
  arXiv  Detail & Related papers  (2022-05-12T17:59:56Z)
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension [51.953428342923885]
  We develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning mechanism is developed to learn the fine difference between the accurate answer and other candidates.
  arXiv  Detail & Related papers  (2021-12-09T07:31:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.