Is Code Better Than Language for Algorithmic Reasoning

• URL: http://arxiv.org/abs/2606.15589v1
• Date: Sun, 14 Jun 2026 04:17:21 GMT
• Title: Is Code Better Than Language for Algorithmic Reasoning
• Abstract summary: We show that for tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism.<n>We separate these factors with an intermediate intervention: the model expresses its reasoning as executable code, and the language model simulates that code in context to produce an answer.<n>We observe that the intermediate intervention is not meaningfully different from natural-language reasoning (+0.15pp)
• Score: 58.86873062890482
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism. We separate these factors with an intermediate intervention: the model expresses its reasoning as executable code, and the language model simulates that code in context to produce an answer. On a 40-task verifiable algorithmic benchmark, deterministic code execution outperforms natural-language reasoning by +31.6pp. We observe that the intermediate intervention is not meaningfully different from natural-language reasoning (+0.15pp). These results suggest that, in our evaluated setting, changing the intermediate representation alone does not explain the tool-use advantage, providing evidence for the performance gains requiring reliable external execution. We formalize this intuition with a simple statistical decision-theoretic model that characterizes when execution dominates end-to-end risk in our disentangled trace-generation/execution regime. We validate our theory using a reconstruction intervention that leverages a proxy language model to infer natural-language reasoning traces from code representations, recovering performance comparable to the original natural-language reasoning pipeline. All experiments are at https://github.com/TerryTong-Git/ToolProj.

Related papers

• The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis [50.473217152006875]
  MIMIC is a framework that leverages executable code as a rigorous medium for reasoning data synthesis.<n>MIMIC transforms algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation.<n>Our method significantly elevates accuracy across general reasoning, complex mathematical benchmarks, and fine-grained deterministic tasks.
  arXiv  Detail & Related papers  (2026-09-13T17:04:16Z)
• Forethought: Verifiable Reasoning from Neurosymbolic Primitive Programming [0.764671395172401]
  Forethought is a neurosymbolic reasoning system that treats reasoning as an explicit, verifiable program.<n>It improves base-model accuracy by about 30% relative and outperforms vanilla prompting, reinforcement learning scaffolds, and prompt-evolution methods.
  arXiv  Detail & Related papers  (2026-07-05T03:20:23Z)
• Do Sparse Autoencoders Identify Reasoning Features in Language Models? [12.693974363520423]
  We investigate whether sparse autoencoders (SAEs) identify genuine reasoning features in large language models (LLMs)<n>We first show through a simple theoretical analysis that $ell_$-regularized SAEs are intrinsically biased toward low-dimensional patterns.<n>Motivated by this bias, we introduce a falsification-oriented evaluation framework to test whether feature activation reflects reasoning processes or superficial linguistic correlates.
  arXiv  Detail & Related papers  (2026-01-09T09:54:36Z)
• Generating Verifiable CoT from Execution-Traces [6.634229408414094]
  Chain-of-Thought prompting has shown promise, but current synthetic training data suffers from a critical weakness.<n>We address this by grounding CoT generation directly in program execution traces.<n>This execution-grounded approach ensures every reasoning step reflects what the program genuinely computes.
  arXiv  Detail & Related papers  (2025-11-28T07:43:43Z)
• Are Language Models Efficient Reasoners? A Perspective from Logic Programming [109.47572890883248]
  Modern language models (LMs) exhibit strong deductive reasoning capabilities, yet standard evaluations emphasize correctness while overlooking a key aspect of human-like reasoning: efficiency.<n>We propose a framework for assessing LM reasoning efficiency through the lens of logic programming.
  arXiv  Detail & Related papers  (2025-10-29T15:30:31Z)
• DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models [58.439517684779936]
  This paper proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way.<n>To ensure a more reliable evaluation, we also introduce a new evaluation metric that mitigates the influence of bias and randomness inherent in Large Language Models.
  arXiv  Detail & Related papers  (2025-09-19T04:40:46Z)
• Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study [37.52166353495979]
  This work investigates how reasoning rules can be explicitly embedded and memorised within language models.<n>We propose a complete pipeline for learning reasoning rules within Transformer-based language VAEs.
  arXiv  Detail & Related papers  (2025-06-24T08:38:03Z)
• Preventing Language Models From Hiding Their Reasoning [0.0]
  Large language models (LLMs) often benefit from intermediate steps of reasoning to generate answers to complex problems. In this work, we focus on one potential way intermediate steps of reasoning could be unfaithful: encoded reasoning. We show that language models can be trained to make use of encoded reasoning to get higher performance without the user understanding the intermediate steps of reasoning.
  arXiv  Detail & Related papers  (2023-10-27T22:02:29Z)
• Rationale-Augmented Ensembles in Language Models [53.45015291520658]
  We reconsider rationale-augmented prompting for few-shot in-context learning. We identify rationale sampling in the output space as the key component to robustly improve performance. We demonstrate that rationale-augmented ensembles achieve more accurate and interpretable results than existing prompting approaches.
  arXiv  Detail & Related papers  (2022-07-02T06:20:57Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.