Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts

• URL: http://arxiv.org/abs/2606.23175v1
• Date: Mon, 22 Jun 2026 11:14:07 GMT
• Title: Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts
• Abstract summary: Coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making.<n>A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases.<n>A companion recomputation flags the remaining cases when the correct observable is known.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argues that outcome-only evaluation is insufficient, and that task outcome, mechanism fidelity, and epistemic honesty must be measured separately. Our evidence comes from 28 episodes of a coding agent attempting to rediscover a known particle identification observable in a Geant4 simulation, including an 8-episode probe across two additional frontier models. In 4/20 primary-model and 3/8 cross-model episodes, agents reach right-looking results through incorrect reasoning that breaks when conditions change, which we call Correct Answer, Wrong Mechanism (CAWM). Honesty and mechanism fidelity dissociate within a single agent trajectory. When given a partially misleading prior, all five agents reject the false component on evidence, yet one defends its chosen observable with physics inconsistent with its own data. In the simulation-based discovery setting studied here, coding agents prove reliable tools but unreliable scientific co-authors for open-ended claim-making, where co-author trust requires mechanism-fidelity verification they do not reliably self-apply. The failure is detectable, and we propose a lightweight test. A one-step regime-shift check needs only the agent's claim and flags the over-generalized cases. A companion recomputation flags the remaining cases when the correct observable is known. Together, these checks flag every CAWM case in this study.

Related papers

• TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents [11.367029696505357]
  We present TruthInsightBench, a benchmark configured for discovery.<n>Its 40 blind tasks, drawn from 40 peer-reviewed studies across 10 scientific domains, expose only a neutral scientific objective and frozen data.<n>A fixed LLM-based judge scores the evidentiary maturity of an agent's own claims along six dimensions.
  arXiv  Detail & Related papers  (2026-09-04T12:37:20Z)
• EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards [73.12811119273206]
  We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations.<n>Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer.<n>Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%.
  arXiv  Detail & Related papers  (2026-08-24T17:29:16Z)
• TRACES: A Benchmark for Epistemic Reliability in Scientific Reasoning by LLMs [0.0]
  Large language models are being proposed as agents in scientific, in domains where no downstream verifier exists.<n>We introduce a probe corpus of 42 retracted, fraudulent, and pseudoscientific papers, paired with a methodology for eliciting and scoring single-shot model engagement with each paper's framing.<n>The probes span five claim types: fabricated observation, pseudophysical mechanism, magical premise, legitimization bridge, and cargo-cult experiment.
  arXiv  Detail & Related papers  (2026-08-11T20:30:32Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• HypoAgent: An Agentic Framework for Interactive Abductive Hypothesis Generation over Knowledge Graphs [52.91976391891331]
  Abductive reasoning over knowledge graphs aims to generate logical hypotheses that explain observed entities or facts.<n>Existing controllable hypothesis generation methods allow users to guide this process with explicit conditions.<n>We propose HypoAgent, an Agentic framework for interactive abductive Hypothesis Generation over knowledge graphs.
  arXiv  Detail & Related papers  (2026-05-29T14:40:37Z)
• When Should an AI Scientist Stop? Verifiable Experiment Steering and Refusal for Autonomous Discovery [0.0]
  CARTOGRAPH is a verification layer for AI scientists.<n>It couples unresolved-subspace experiment steering (select), explicit ambiguity closure (resolve), and residual-based library inadequacy detection (refuse)
  arXiv  Detail & Related papers  (2026-05-26T18:19:16Z)
• CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists [28.253879252786632]
  We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents.<n>Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism.
  arXiv  Detail & Related papers  (2026-05-25T16:57:06Z)
• A measurement substrate for agentic Kubernetes operations: Methodology and a case study in retrieval-compounding falsification [0.0]
  Empirical claims about autonomous operations agents are largely unfalsifiable.<n>Code agents have a verification substrate that turns "did it work" into a fast, falsifiable, groundtruth signal.<n>We present agent-breakage, a closed-loop measurement framework that injects faults into a target cluster.
  arXiv  Detail & Related papers  (2026-05-21T21:47:52Z)
• VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems [79.51005192758262]
  Large language model-driven multi-agent systems excel at complex tasks.<n>However, unreliable agents remain a key bottleneck to system-level reliability.<n>We propose a hypothesis verification framework for agent failure attribution.
  arXiv  Detail & Related papers  (2026-05-17T14:09:35Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Decoding the Critique Mechanism in Large Reasoning Models [50.821607345799386]
  Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions.<n>We investigate how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps.<n>We find that despite the error propagating through the chain-of-thought, the model still reaches the correct final answer.
  arXiv  Detail & Related papers  (2026-03-17T10:03:30Z)
• To Throw a Stone with Six Birds: On Agents and Agenthood [0.0]
  Six Birds Theory (SBT) treats macroscopic objects as induced closures rather than primitives.<n>We give a type-correct account of agency within SBT.<n>We operationalize this contract in finite controlled systems using four checkable components.
  arXiv  Detail & Related papers  (2026-02-03T10:46:23Z)
• AI-Assisted Engineering Should Track the Epistemic Status and Temporal Validity of Architectural Decisions [0.0]
  LLM coding assistants generate decisions faster than teams can validate them.<n>No widely-adopted framework distinguishes conjecture from verified knowledge.<n>We propose three requirements for responsible AI-assisted engineering.
  arXiv  Detail & Related papers  (2026-01-28T23:12:07Z)
• Missci: Reconstructing Fallacies in Misrepresented Science [84.32990746227385]
  Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Missci is a novel argumentation theoretical model for fallacious reasoning. We present Missci as a dataset to test the critical reasoning abilities of large language models.
  arXiv  Detail & Related papers  (2024-06-05T12:11:10Z)
• Nested Counterfactual Identification from Arbitrary Surrogate Experiments [95.48089725859298]
  We study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. Specifically, we prove the counterfactual unnesting theorem (CUT), which allows one to map arbitrary nested counterfactuals to unnested ones.
  arXiv  Detail & Related papers  (2021-07-07T12:51:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.