Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness

• URL: http://arxiv.org/abs/2606.11686v1
• Date: Wed, 10 Jun 2026 05:55:13 GMT
• Title: Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness
• Abstract summary: A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
• Score: 45.148328075418156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: End-to-end task-success is the dominant way to evaluate LLM agents, but one aggregate number tells you that an agent regressed, not where. We present layer-isolated evaluation: a deployed ordering agent is decomposed into a fixed taxonomy of layers (ontology, intent, routing, decomposition, escalation, safety, memory, and cross-cutting envelope/defense), each exercised by its own assertion slice in a deterministic, no-LLM "pure" mode. The pure suite (238 cases across 23 slices; 225 run in 2.39 s, ~10 ms/case) runs in CI on every change against a locked per-slice baseline. We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers. The effect we did not design in is masking: the aggregate pass-rate barely moves (-1.7 to -5.9 pp for six local regressions), while the matching slice craters (-25 to -91 pp). A layer's slice reacting to its own fault is partly by construction; the measured results are (i) the aggregate masking and (ii) that damage stays off the other slices: the injected layer's slice is the single worst-hit in 5 of 7 cases and top-3 in 7 of 7 (mean rank 1.29 of 19). Localization replicates on a second, structurally different tenant (Starbucks SG): all seven matching slices crater, so it is not a single-catalog artifact. We position it as a concrete, deterministic instantiation of the component-level evaluation EDDOps prescribes but leaves unimplemented, with CheckList as ancestor and as the deterministic mirror image of whole-workflow stochastic mutation testing. Our contributions: (a) a fully decomposed, sub-second, no-LLM per-layer harness for a production agent, (b) a coverage-honesty test-adequacy criterion that refuses to score an unexercised layer, and (c) the regression-injection demonstration that per-slice baseline-locked gates localize regressions an aggregate metric masks.

Related papers

• SiLR: Structure-Preserving Admission and Process Reward for LLM Tool Agents [5.864346468963738]
  We study post-violation recovery admission, where progress must be admitted while the system is still in violation.<n>SiLR shadow-executes each proposal and admits it under a product order over the branch-level violation state.<n>We prove that no scalar surrogate is sound for this order, so the failure is representational, not a matter of threshold tuning.
  arXiv  Detail & Related papers  (2026-09-04T02:03:52Z)
• VESTIGE: A Knowledge-Guided Masking Strategy for Corruption-Aware Fine-Tuning of Genomic Transformers, Validated on Ancient DNA Reconstruction [0.0]
  Masked--model fine-tuning applies a uniform masking probability across every token position, assuming reconstruction difficulty is position-agnostic.<n>We introduce VESTIGE, a parameter-free, drop-in replacement for the standard collator that aligns the masking distribution with an empirically measured per-position corruption profile.
  arXiv  Detail & Related papers  (2026-07-30T05:47:47Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• SkillDAG: Self-Evolving Typed Skill Graphs for LLM Skill Selection at Scale [54.70985426016736]
  We present SkillDAG, which models inter-skill relationships as a typed directed graph.<n>Each search returns vector matches, typed-edge neighbors, and conflict signals, and a propose-then-commit protocol lets the agent register execution-backed edges.<n>On ALFWorld and SkillsBench with MiniMax-M2.7, SkillDAG reaches 67.1% success and 27.3% reward.
  arXiv  Detail & Related papers  (2026-06-02T02:45:21Z)
• Token-Level Generalization in LoRA Adapter Backdoors: Attack Characterization and Behavioral Detection [0.0]
  We show that LoRA can be reliably backdoored through training data poisoning.<n>The attack scales monotonically with rank, and the chosen trigger-anchor token is both trigger-dependent and base-model-dependent.<n> Behavioral detection is the operationally portable result for supply chain scanning.
  arXiv  Detail & Related papers  (2026-05-28T16:32:25Z)
• Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection [0.0]
  Multi-agent debate improves factuality and reasoning, but most recipes pick a fixed round count.<n>We adapt Wald's Sequential Probability Ratio Test (SPRT) as a plug-in compute governor for LLM debates.<n>On GSM8K the rule stops in 1.01 average rounds (4.06 LLM calls) at 97.0% accuracy vs 99.0% for fixed-5 debate at 15 calls: a 3.7x call reduction at -2pp accuracy.<n>On MMLU the calibrated KL collapses to about 0 and the rule caps on 99.5% of items at 2.1x cost.
  arXiv  Detail & Related papers  (2026-05-18T23:43:12Z)
• Not Just RLHF: Why Alignment Alone Won't Fix Multi-Agent Sycophancy [0.7212939068975618]
  We find that pretrained base models exhibit the same substitution pattern as their Instruct variants, averaging higher yield than Instruct.<n> patching above this window restores 96% of the clean-to-pressured P(correct) gap.<n>Two converging activation-space interventions show that pressure suppresses clean-reasoning features rather than activating a new sycophancy circuit.
  arXiv  Detail & Related papers  (2026-05-13T04:45:08Z)
• GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives [48.545980031973556]
  GAMBIT is a benchmark with three evaluation modes and two independent scores for evaluating imposter detectors.<n>The benchmark comes with a dataset of 27,804 labeled instances spanning 240 co-evolved imposter strategies.
  arXiv  Detail & Related papers  (2026-05-09T16:07:23Z)
• Partial Number Theoretic Transform Masking in Post Quantum Cryptography Hardware: A Security Margin Analysis [0.0]
  Adams Bridge, a hardware accelerator for ML-DSA and ML-KEM, masks 1 of its Inverse Number Theoretic Transform layers and relies on shuffling for the remainder.<n>We evaluate these claims against published side-channel literature across seven analysis tracks with confidence-rated evidence.
  arXiv  Detail & Related papers  (2026-04-04T17:38:34Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• Quantile Advantage Estimation for Entropy-Safe Reasoning [44.192277495613695]
  Reinforcement Learning with Verifiable Rewards (RLVR) strengthens LLM reasoning, but training often oscillates between entropy collapse and entropy explosion<n>We trace both hazards to the mean baseline used in value-free RL, which improperly penalizes negative-advantage samples under reward outliers.<n>We propose Quantile Advantage Estimation (QAE), replacing the mean with a group-wise K-quantile baseline.
  arXiv  Detail & Related papers  (2025-09-26T17:37:52Z)
• Void in Language Models [2.7195102129095003]
  We analyze layer activation in instruction-tuned language models (LMs) across two phases: Prompt Processing and Response Generation.<n>We show that not all layers contribute equally during inference, and that selectively skipping most of them can improve the performance of models on certain tasks.
  arXiv  Detail & Related papers  (2025-05-20T15:01:56Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [67.76186488361685]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.