Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop

• URL: http://arxiv.org/abs/2607.23002v1
• Date: Sat, 25 Jul 2026 02:38:07 GMT
• Title: Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop
• Abstract summary: We study an adversarial test-hardening loop under a mechanical oracle.<n>A Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those.<n>We release both protocols, all receipts, and the analysis code.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models increasingly write both code and the tests meant to check it; coverage records what ran, not what was verified. We study an adversarial test-hardening loop under a mechanical oracle: a Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those, with every verdict decided mechanically, so no model judges another's output. In Experiment 1, on five Python subjects (one same-lineage-loop cell could not be scored), the loop killed 105 mutants that one-shot generation missed and lost none, and the cross-lineage-Critic question returned a pre-declared null. The central finding was an autopsy: an earlier analysis reported a cross-lineage effect at p = 9.5e-66 that was an instrument artifact, an output cap silently truncating the verbose model, caught only by adversarial review of the completed analysis. Review then found a further confound, each arm resampling its own initial suite; Experiment 2 removes it. Under a pre-registered frozen-shared-round-0 design (five replicates on each of four subjects, seeds committed in advance), same-lineage Critic rounds killed 78% of the survivors the frozen initial suite left standing (mean incremental kill rate 0.783, 95% cluster-bootstrap interval [0.592, 0.935]), a within-replicate causal estimate; the cross-provider configuration showed a positive pilot difference (rate gap 0.178, 95% interval [0.039, 0.347]; magnitude dominated by a single replicate) at 5.5x lower arm cost. This compares two named model-provider-harness configurations, not an isolated lineage effect: part of the gap is one configuration's receipted operational failures, including truncation recurrences, now detected and scored rather than laundered. Cross-model comparisons can inherit the asymmetries of the harness that runs them. We release both protocols, all receipts, and the analysis code.

Related papers

• The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent [0.0]
  We evaluate whether a verifier-and-acceptance stage - a model verifier whose verdicts are enforced by deterministic code - changes what an LLM-driven offensive-security agent reports.<n>We report a 15-run exploratory pilot, a pre-registered 20-run confirmatory ablation, and a pre-registered 2 x 2 factorial study with 40 runs across two deliberately vulnerable lab targets.<n>The results support a narrow conclusion: the verifier changes what the system ships, while deterministic code supplies enforcement and auditability.
  arXiv  Detail & Related papers  (2026-09-14T17:13:05Z)
• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints [4.41537705949485]
  Language-model judges now gate training data, score generations, and drive leaderboards.<n>We audited that assumption in two preregistered campaigns with every threshold fixed in advance.<n>Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.<n>We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist.
  arXiv  Detail & Related papers  (2026-09-03T17:59:43Z)
• The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision-Language Models [22.461983628330277]
  Many few-shot adaptation methods classify with a convex combination of the zero-shot text prototype and the mean of the K labelled image features.<n>We ask what is the right ratio, can it be estimated without validation data, and is finding it where the performance is?<n>Across 4,800 cells (ten datasets, five backbones including SigLIP, five shot counts, five seeds, four prompt tiers) this theoretically optimal ratio is a reliable estimate of the wrong quantity.<n> validation-free linear probes beat even the oracle-tuned blend: CLAP by +1.9 points and LP++ by +1.5 on average.
  arXiv  Detail & Related papers  (2026-08-23T15:02:33Z)
• Agent Behavioral Contracts II: Certifying Compositional Reliability Without Assuming Independence [0.0]
  Two instances of one model, in a two-agent handoff, co-fail on 90.0% of the missions on which either fails.<n>Substituting a different model reduces the association in six of six contrasts.
  arXiv  Detail & Related papers  (2026-08-13T07:25:05Z)
• Excess Separability: Nuisance-Controlled Residual-Stream Probing for Benchmark Contamination Detection [0.0]
  Benchmark contamination is diagnosed with n-gram overlap, likelihood-based membership inference, or canary strings.<n>We show the natural way to do this does not work, specify one that survives measurement, then find that the correction making it work carries more variance than the null it is tested against.
  arXiv  Detail & Related papers  (2026-08-12T23:27:20Z)
• Caliber: Cross-Architecture Extraction-Cost Control for Score-Returning APIs [27.21644231867863]
  Caliber is a defense against model extraction that formulates noise selection as a calibration problem.<n>To defend against an attacker that uses returned scores for knowledge distillation, Caliber adds independent and identically distributed Gaussian noise to the internal logits.<n>Caliber normalizes noise variance by the squared median top-two logit margin and fits the resulting noise-utility relationship with a logistic curve.
  arXiv  Detail & Related papers  (2026-08-02T05:58:00Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• Retrying vs Resampling in AI Control [0.42970700836450476]
  We study retrying from an AI control perspective, which treats the model as potentially adversarial.<n>We find that while retrying reduces honest suspicion scores, the untrusted model can exploit monitor rationale to construct sneakier attacks.
  arXiv  Detail & Related papers  (2026-05-25T17:10:41Z)
• When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation [9.055086193088083]
  We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
  arXiv  Detail & Related papers  (2026-05-25T15:57:11Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes [24.516406575400016]
  We study auto research as a closed empirical loop driven by external measurement.<n>Each submitted trial carries a hypothesis, an executable code edit, an evaluator-owned outcome, and feedback that shapes the next proposal.<n>We instantiate this loop with specialist agents that partition recipe surfaces and share measured lineage across trials.
  arXiv  Detail & Related papers  (2026-05-07T06:13:43Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• RULSurv: A probabilistic survival-based method for early censoring-aware prediction of remaining useful life in ball bearings [39.58317527488534]
  We introduce a novel and flexible method for early fault detection using Kullback-Leibler divergence and RUL estimation.<n>We demonstrate our approach in the XJTU-SY dataset using a 5-fold cross-validation strategy across three different operating conditions.<n>Our approach achieves a mean cumulative relative accuracy (CRA) of 0.7586 over 5 bearings under the highest load, which improves over several state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-05-02T16:17:29Z)
• Predicting Survival Time of Ball Bearings in the Presence of Censoring [44.99833362998488]
  We propose a novel approach to predict the time to failure in ball bearings using survival analysis. We analyze bearing data in the frequency domain and annotate when a bearing fails by comparing the Kullback-Leibler divergence and the standard deviation. We train several survival models to estimate the time to failure based on the annotated data.
  arXiv  Detail & Related papers  (2023-09-13T08:30:31Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.