Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling

• URL: http://arxiv.org/abs/2608.23653v1
• Date: Mon, 24 Aug 2026 11:50:07 GMT
• Title: Beyond Executable Models: The Pufibara Agent Harness and the Modelica Agent Workflow Benchmark for Physical System Modeling
• Abstract summary: Physical system modeling presents different requirements from general-purpose code generation in software engineering.<n>We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements.<n>We present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution simulation evidence with the candidate that produced it, and makes submission an explicit agent action.
• Score: 3.2102307224389484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents are increasingly used for simulation-driven engineering. Physical system modeling presents different requirements from general-purpose code generation in software engineering, because correctness depends not only on syntax and executability but also on physical consistency and scenario-dependent behavior. We study this challenge in Modelica, an equation-based modeling language in which a model may compile and simulate while still violating its intended physics or engineering requirements. Across successive revisions, an agent may lose track of requirements or rely on simulation evidence produced by an outdated candidate. To address this challenge, we present Pufibara, an agent harness that maintains persistent engineering state across revisions, associates execution and simulation evidence with the candidate that produced it, and makes submission an explicit agent action. To evaluate end-to-end Modelica agent workflows, we also propose a source-grounded method for constructing realistic and independently evaluable tasks. We use this method to build the 232-task Modelica Agent Workflow Benchmark, spanning Model Repair, Model Generation, and Model Tuning. Each submitted candidate is scored by a benchmark-owned evaluator outside the agent loop. We compare Pufibara with Claude Code as complete harnesses under two matched large language model (LLM) backends. With DeepSeek v4 Flash, Pufibara passes 202 tasks, compared with 185 for Claude Code. With Claude Sonnet 5, Pufibara passes 202 tasks, compared with 187 for Claude Code. Under the repository-reported token accounting, Pufibara records 76.4%-82.5% lower logical-token totals. Its sequential runtime is 6.1%-58.4% lower. These findings show that, even under matched LLM backends, complete agent harnesses can differ substantially in both task success and resource use for physical system modeling.

Related papers

• MADE: Belief-Driven Dual-Agent Coordination for Autonomous Model Deployment [19.126846876843356]
  LLM-based agents struggle with domain-specific tasks, motivating the integration of external tools.<n>Open-source community offers vast array of AI models typically released as heterogeneous research artifacts.<n>Transforming these into ready-to-call APIs is costly and labor-intensive.<n>We introduce Model Automated Deployment Engine (MADE), a dual-agent coordination system.
  arXiv  Detail & Related papers  (2026-08-02T12:14:43Z)
• KAT-Coder-V2.5 Technical Report [61.907486544595336]
  KAT-Coder-V2.5 is a coding-focused agentic model trained to act autonomously inside real, executable repositories.<n>It reconstructs multilingual repositories into sandboxed environments with fail-to-pass and pass-to-pass verification at scale.<n>We further scale reinforcement learning with harness randomization, a reliability-hardened sandbox, an asymmetric actor--critic PPO with hindsight-augmented value estimation, and a harness-oriented reward framework.
  arXiv  Detail & Related papers  (2026-07-06T08:14:02Z)
• Dissecting model behavior through agent trajectories [15.811597127775812]
  We formalize the intent-execution' gap: the mismatch between what the model intends and what the harness executes, and vice versa.<n>To illustrate the impact of this harness-model alignment, we develop a simple and customizable harness called Simple Strands Agent' (SSA)<n>We make two contributions: (i) we reproduce or improve on the pass@1 performance reported by diverse model-provider families on popular agentic benchmarks, and (ii) building on an analysis of 138k trajectories generated by SSA.
  arXiv  Detail & Related papers  (2026-06-16T03:17:03Z)
• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures [1.0742675209112622]
  We study failure modes as stemming from three mechanisms: underspecification, capability errors, and agent harness errors.<n>We propose ClayBuddy, a harness that molds to user preferences and can be modified by the model in-session to mitigate these errors.
  arXiv  Detail & Related papers  (2026-06-13T18:29:53Z)
• Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows [18.6534256358905]
  We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent systems.<n>The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns.<n>Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior.
  arXiv  Detail & Related papers  (2026-05-27T03:47:35Z)
• LLM-driven design of physics-constrained constitutive models: two agents are better than one [0.0]
  We introduce the first multi-agent LLM-driven approach for model generation.<n>A Creator agent proposes a model tailored to the data, while an Inspector agent audits each proposal against nine physical constraints.<n>We demonstrate this concept with artificial neural networks (CANNs) and benchmark it on brain tissue, experimental rubber, and synthetic rubber.
  arXiv  Detail & Related papers  (2026-05-22T15:27:09Z)
• ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generation [55.947962672433675]
  ChipMATE is the first self-trained multi-agent framework for RTL generation.<n>Inspired by industrial practice, ChipMATE pairs a Verilog agent with a Python reference-model agent that mutually verify each other's outputs.<n>ChipMATE achieves 75.0% and 80.1% pass@1 on VerilogEval V2 with 4B and 9B base models.
  arXiv  Detail & Related papers  (2026-05-13T01:04:21Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Model-driven realization of IDTA submodel specifications: The good, the bad, the incompatible? [49.60138105915326]
  Asset Administration Shells are trending in Industry 4.0. In February 2024, the Industrial Digital Twin Association announced 84 and released 18 AAS submodel specifications. We present a model-driven approach, which transforms extracted information from IDTA specifications into an intermediary meta-model and, from there, generates API code and tests.
  arXiv  Detail & Related papers  (2024-06-20T16:33:46Z)
• Model Reuse with Reduced Kernel Mean Embedding Specification [70.044322798187]
  We present a two-phase framework for finding helpful models for a current application. In the upload phase, when a model is uploading into the pool, we construct a reduced kernel mean embedding (RKME) as a specification for the model. Then in the deployment phase, the relatedness of the current task and pre-trained models will be measured based on the value of the RKME specification.
  arXiv  Detail & Related papers  (2020-01-20T15:15:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.