Towards Evaluation of Implicit Software World Models in Coding LLMs

• URL: http://arxiv.org/abs/2606.27406v1
• Date: Thu, 25 Jun 2026 08:02:32 GMT
• Title: Towards Evaluation of Implicit Software World Models in Coding LLMs
• Abstract summary: We take a step toward a broader evaluation by shifting the observable axis to execution resources.<n>We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks.
• Score: 3.4011650749358786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.

Related papers

• The Imitation Game: When LLMs Learn to Reason Like Programs via Code-Centric Reasoning Data Synthesis [50.473217152006875]
  MIMIC is a framework that leverages executable code as a rigorous medium for reasoning data synthesis.<n>MIMIC transforms algorithms into verifiable reasoning trajectories through narrative fusion, code-guided test synthesis, and dynamic code instrumentation.<n>Our method significantly elevates accuracy across general reasoning, complex mathematical benchmarks, and fine-grained deterministic tasks.
  arXiv  Detail & Related papers  (2026-09-13T17:04:16Z)
• Software Aging in LLM-Generated Applications: Runtime Evidence, Static Analysis, and Human-Written Comparisons [2.051924013527311]
  Large Language Models (LLMs) are increasingly used to generate executable software systems from natural language specifications.<n>We experimentally investigate software aging symptoms in LLM-generated service-based applications across generation-and-execution environments.
  arXiv  Detail & Related papers  (2026-08-26T20:34:06Z)
• Latent Programming Horizons in Coding Agents [6.8463578914435645]
  coding agent solving a software-engineering task spends dozens of steps reasoning, editing code, and running tests.<n>We show that the residual streams of language models under coding agents linearly encode properties of the evolving program.<n> Probes trained to predict the outcome of future edits achieve performance above chance up to roughly 25 steps in advance.
  arXiv  Detail & Related papers  (2026-07-06T15:08:26Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• You Don't Need Public Tests to Generate Correct Code [0.9668407688201359]
  We show that large language models possess the capability to autonomously construct valid inputs and simulate execution flows for self-correction.<n>We introduce DryRUN, a framework that removes the necessity for ground-truth data by enabling the LLM to iteratively plan, synthesize its own test inputs, and run simulated executions.
  arXiv  Detail & Related papers  (2026-04-23T12:21:03Z)
• StatsClaw: An AI-Collaborative Workflow for Statistical Software Development [1.8686116192307898]
  Existing AI code-generation tools produce code quickly but cannot guarantee faithful implementation.<n>We introduce StatsClaw, a multi-agent architecture for Claude Code that enforces information barriers between code generation and validation.<n>Results show that StatsClaw can absorb the engineering overhead of the software lifecycle while preserving researcher control over every substantive methodological decision.
  arXiv  Detail & Related papers  (2026-04-06T17:18:53Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• From Benchmark Data To Applicable Program Repair: An Experience Report [1.6913109767046948]
  This paper describes our approach to automated program repair.<n>We combine various techniques from the literature to achieve this.<n>Experiments show that our approach performs better than other techniques on standard benchmarks.<n>On closer inspection, none of these techniques work on realistic defects that we see in industry.
  arXiv  Detail & Related papers  (2025-08-22T03:59:27Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• SURGE: On the Potential of Large Language Models as General-Purpose Surrogate Code Executors [7.210032327838313]
  We investigate whether large language models (LLMs) can serve as surrogate models for code execution prediction.<n>Through extensive analysis of $21$ open-source and proprietary LLMs, we examine scaling laws, data efficiency, and predictive accuracy.<n>Our findings reveal important insights about the feasibility of LLMs as efficient surrogates for computational processes.
  arXiv  Detail & Related papers  (2025-02-16T15:38:19Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.