EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

• URL: http://arxiv.org/abs/2603.09678v1
• Date: Tue, 10 Mar 2026 13:47:15 GMT
• Title: EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages
• Abstract summary: Large language models achieve near-ceiling performance on code generation benchmarks, yet these results increasingly reflect memorization rather than genuine reasoning.<n>We introduce EsoLang-Bench, a benchmark using five esoteric programming languages that lack benchmark gaming incentives due to their economic irrationality for pre-training.<n>These languages require the same computational primitives as mainstream programming but have 1,000-100,000x fewer public repositories than Python.
• Score: 5.37133760455631
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models achieve near-ceiling performance on code generation benchmarks, yet these results increasingly reflect memorization rather than genuine reasoning. We introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare) that lack benchmark gaming incentives due to their economic irrationality for pre-training. These languages require the same computational primitives as mainstream programming but have 1,000-100,000x fewer public repositories than Python (based on GitHub search counts). We evaluate five frontier models across five prompting strategies and find a dramatic capability gap: models achieving 85-95% on standard benchmarks score only 0-11% on equivalent esoteric tasks, with 0% accuracy beyond the Easy tier. Few-shot learning and self-reflection fail to improve performance, suggesting these techniques exploit training priors rather than enabling genuine learning. EsoLang-Bench provides the first benchmark designed to mimic human learning by acquiring new languages through documentation, interpreter feedback, and iterative experimentation, measuring transferable reasoning skills resistant to data contamination.

Related papers

• Bootstrapping Niche Multilingual Code Translation via Reinforcement Learning with Execution-Based Verifiable Supervision [41.68344013253776]
  We introduce HumanEval-X++, an execution-based benchmark that extends HumanEval-X to a broad many-to-many language space.<n>On HumanEval-X++ and existing benchmarks, it yields consistent gains over the untrained baselines.<n>Our study establishes a reliable approach of data generation, training, and benchmarking, paving the way toward further bootstrapping the quality of many-to-many translation for programming languages.
  arXiv  Detail & Related papers  (2026-08-14T00:56:41Z)
• Syntax Without Semantics: Teaching Large Language Models to Code in an Unseen Language [9.529147118376464]
  Python outperforms PyLang by up to 19% across all configurations.<n>Models possess language-agnostic understanding but cannot express it in an unfamiliar language.
  arXiv  Detail & Related papers  (2026-05-15T04:37:31Z)
• Selecting Language Models for Social Science: Start Small, Start Open, and Validate [0.3823356975862005]
  We argue that social scientists cannot altogether avoid validating computational measures (ex-post)<n>Being able to reliably replicate a particular finding that entails the use of a language model requires reliably reproducing a task.<n>We propose starting with smaller, open models, and constructing benchmarks to demonstrate the validity of the entire computational pipeline.
  arXiv  Detail & Related papers  (2026-01-16T01:01:47Z)
• Anka: A Domain-Specific Language for Reliable LLM Code Generation [0.0]
  Large Language Models (LLMs) exhibit systematic errors on complex, multi-step programming tasks.<n>We introduce Anka, a domain-specific language () for data transformation pipelines designed with explicit, constrained syntax.<n>Anka achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems.
  arXiv  Detail & Related papers  (2025-12-29T05:28:17Z)
• UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian [0.42970700836450487]
  This paper introduces UA-Code-Bench, a new open-source benchmark established for a thorough evaluation of language models' code generation and competitive programming problem-solving abilities in Ukrainian.<n>The benchmark comprises 500 problems from the Eolymp platform, evenly distributed across five complexity levels from very easy to very hard.<n>The obtained results reveal that even top-performing models, such as OpenAI o3 and GPT-5, solve only half of the problems.
  arXiv  Detail & Related papers  (2025-11-07T07:24:56Z)
• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.1875460416205]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages.<n>It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total.<n>Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• XferBench: a Data-Driven Benchmark for Emergent Language [1.6574413179773761]
  We introduce a benchmark for evaluating the overall quality of emergent languages. We measure this by using the emergent language as pretraining data for a downstream NLP tasks in human language. We empirically test the benchmark's validity using human, synthetic, and emergent language baselines.
  arXiv  Detail & Related papers  (2024-07-03T19:02:26Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• Multi-lingual Evaluation of Code Generation Models [82.7357812992118]
  We present new benchmarks on evaluation code generation models: MBXP and Multilingual HumanEval, and MathQA-X. These datasets cover over 10 programming languages. We are able to assess the performance of code generation models in a multi-lingual fashion.
  arXiv  Detail & Related papers  (2022-10-26T17:17:06Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.