Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software

• URL: http://arxiv.org/abs/2605.30353v1
• Date: Thu, 28 May 2026 17:59:59 GMT
• Title: Physics Is All You Need? A Case Study in Physicist-Supervised AI Development of Scientific Software
• Abstract summary: A physicist supervised an AI coding agent over 12 work days and 57 sessions to build CLAX-PT.<n>The agent resolved ten autonomously by iterating against oracle tests.<n>The three it could not -- all evaded oracle detection -- share a common property: the agent treated symptom reduction as root-cause resolution.
• Score: 0.304585143845864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Are AI agents tools, co-authors, or researchers? We present a quantified case study ($N=1$): a physicist supervising an AI coding agent (Claude Code, Sonnet and Opus models) over 12 work days and 57 sessions to build CLAX-PT, a differentiable one-loop perturbation theory module in JAX. We documented and classified 15 supervision events by intervention level. The agent resolved ten autonomously by iterating against oracle tests. Two more by the physicist's domain knowledge. The three it could not -- all evaded oracle detection -- share a common property: the agent treated symptom reduction as root-cause resolution. It spent 33 of the 57 sessions adjusting coefficients within a code architecture that could not represent the target physics, and could not re-evaluate its CLASS-PT branch choice even when prompted to reconsider; only an injected physics concept (anisotropic BAO damping) triggered the redesign. Separately, the agent committed a calibrated correction that passed all oracle tests but corresponded to no quantity in the theory, predicting wrong values at any other cosmology. The fudge factor was caught and replaced within the same session. Three supervision practices proved critical for catching what oracle tests missed: testing at diverse parameter points beyond the fiducial calibration; shared changelogs that surfaced stalled exploration across sessions; and an explicit rule against unphysical numerical patches. In this case, supervision design, not model capability, determined whether the agent's output was trustworthy. Closing the gap would require agents that propose architectural alternatives rather than optimize within a given structure, and distinguish predictive adequacy from explanatory correctness -- capabilities not exhibited here, not obviously addressed by scaling alone. [Abridged.]

Related papers

• Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence [86.61561123749011]
  We introduce Apodex Discovery, a framework for building and evaluating discoverative AI through the heavy-duty solver.<n>First, a problem-scouting process surveyed 561 industries across 16 sectors, assembled 423 high-value real-world problems, and selected 20 for the initial release.<n>Second, a common environment-task-episode abstraction provides data, tools, constraints, feedback, trajectory recording, and verification of intermediate artifacts and final submissions.
  arXiv  Detail & Related papers  (2026-08-11T18:48:40Z)
• Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation [1.4180331276028662]
  A/B testing remains the standard for rolling out new features in the technology industry.<n>Can AI agents---conditioned on behavioral profiles and contextual descriptions of the intervention---simulate outcomes accurately enough to vet candidate treatments before committing live traffic?<n>We formalize this question as a emphSimulated Randomized Controlled Trial (S-RCT) and derive a two-layer error decomposition that separates agent approximation error from subsampling error.
  arXiv  Detail & Related papers  (2026-08-03T14:58:06Z)
• 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse [15.948323546029714]
  Coding agents now author entire pull requests.<n> practitioners sharply disagree about what this does to code review.<n>Agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones.
  arXiv  Detail & Related papers  (2026-07-08T23:11:46Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts [0.0]
  Coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making.<n>A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases.<n>A companion recomputation flags the remaining cases when the correct observable is known.
  arXiv  Detail & Related papers  (2026-06-22T11:14:07Z)
• Science Earth: Towards A Planet-Scale Operating System for AI-Native Scientific Discovery [51.70607714288616]
  Science Earth is a planet-scale scientific runtime in which any capability can connect to any other.<n>Its underlying EACN protocol lets capabilities discover one another, negotiate task ownership, and adjudicate across incompatible evidentiary standards.<n>Two runs validate this under structurally distinct conditions.
  arXiv  Detail & Related papers  (2026-05-31T16:05:41Z)
• The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems [0.0]
  This thesis turns impossibility results from curiosities into design rules.<n>Its flagship result proves an accuracy ceiling set by architecture alone.<n>The same argument recasts across subfields.
  arXiv  Detail & Related papers  (2026-05-21T20:48:35Z)
• AI CFD Scientist: Toward Open-Ended Computational Fluid Dynamics Discovery with Physics-Aware AI Agents [1.7590081165362783]
  We present AI CFD Scientist, an open-source AI scientist for computational fluid dynamics.<n>It is the first to span literature-grounded ideation, validated execution, vision-based physics verification, source-code modification, and figure-grounded writing within a single inspectable workflow.
  arXiv  Detail & Related papers  (2026-05-07T17:27:23Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• The Specification as Quality Gate: Three Hypotheses on AI-Assisted Code Review [0.0]
  The dominant industry response to AI-generated code quality problems is to deploy AI reviewers.<n>This paper argues that this response is structurally circular when executable specifications are absent.
  arXiv  Detail & Related papers  (2026-03-26T11:59:05Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL [86.0987896274354]
  We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. We then propose a novel Self-Excite Eigenvalue Measure (SEEM) metric to measure the evolving property of Q-network at training. For the first time, our theory can reliably decide whether the training will diverge at an early stage.
  arXiv  Detail & Related papers  (2023-10-06T17:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.