When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline

• URL: http://arxiv.org/abs/2609.12017v1
• Date: Thu, 10 Sep 2026 07:55:12 GMT
• Title: When Agent Metrics Measure Different Things: An Evidence-Grounded Audit of the Praxa AI Pipeline
• Abstract summary: We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records.<n>A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures.<n>The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent evaluations can be numerically correct while measuring a different construct from the one implied by their labels. We present a retrospective measurement audit of selected Praxa AI implementation files, historical evaluation artifacts, and operational records. A 139-case offline routing report contains 112 passes and 27 failures despite zero gating failures, because known gaps are explicitly exempted from the gate. An identifier-free export represents 8,843 tool-attempt rows: 8,395 recorded durations and 448 missing values. Of the durations, 121 equal the signed 32-bit maximum and carry abandoned-client labels; inspected database code clamps elapsed lifecycle age. The pooled recorded 99th percentile is 2,147,483,647 ms, versus 38,118.31 ms among server-observed completed calls. This is a stratum contrast, not a treatment effect. In a documented single-trajectory compaction pilot, the reported follow-up input reduction is 94.39%, but the reduction across the trigger and follow-up calls together is 46.54%. We reproduce the descriptive calculations, verify 91 timing statistics through a separate weighted rational-arithmetic implementation, and execute 13 scoring-function and 12 analysis-verifier tests. Finite-completion bounds show how missing durations limit all-row timing statements without imputing values. The contribution is a source-linked case study and reusable verification package for separating gate policy, lifecycle timing, and request-level accounting from broader agent-performance claims. Historical provider runs and the full current pipeline were not independently reproduced; general capability superiority and population-level statistical significance are not established.

Related papers

• Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows [0.0]
  Compound AI pipelines chain LLM calls, retrievers, and tools are nondeterministic.<n>We give a protocol covering all three: it commits digests of each stage's inputs, outputs, and context.<n>On a synthetic HotpotQA pipeline a calibrated fixed threshold accepts 44 of 45 honest reproductions and rejects 104 of 105 divergent pairs.
  arXiv  Detail & Related papers  (2026-09-07T22:10:54Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• When May an Agent Stop? Evidence-Carrying Termination for Tool-Using LLMs [1.7936680693145324]
  ECT certifies support in a recorded trace under declared assumptions, not external truth, safety, or alignment.<n>ECT executed successful recovery in 18/66 trajectories, of which 17 subsequently completed with support.
  arXiv  Detail & Related papers  (2026-08-22T18:56:27Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• Does Splitting a Triage Decision Across Agents Hide Bias or Help Catch It? A Multi-Agent Simulation Study of LLM-Based Resource Allocation Under Audit Capacity Constraints [0.0]
  We study what happens to bias when the same decision is distributed across a role-differentiated multi-agent pipeline.<n>We find no measurable difference in how often biased outcomes occur between the two conditions.<n>We do find a large and significant effect of audit capacity on whether bias is caught.
  arXiv  Detail & Related papers  (2026-08-07T08:25:53Z)
• Argus: A General-Purpose Agentic Reasoning Runtime for Long-Horizon Tasks [38.52523278435824]
  We present Argus, a persistent, self-evolving runtime in which Manager, Planner, Engineer, and Reviewer execute bounded missions over durable project state.<n>Argus separates stable user intent from operational objectives, constraints, and verification criteria, and admits memories, skills, procedures, verifiers, routing decisions, and rejected routes only after role-owned review and, when available, task-native verification.
  arXiv  Detail & Related papers  (2026-08-05T17:58:58Z)
• TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows [2.435006380732194]
  We present TraceCompiler, a skill-guided system that compiles noisy traces into trajectories.<n>Every hard edge carries an auditable evidence, and ambiguous relations are marked suspected.<n>We measure call reduction but not offline compilation cost, so we claim no net efficiency result.
  arXiv  Detail & Related papers  (2026-08-03T01:05:06Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• Mean Time to Remediate Is Not a Fielding Model: A Cadence Audit for Enterprise Vulnerability Management [51.56484100374058]
  This paper introduces a remediation-cadence audit for enterprise vulnerability management.<n>It records routine mean lag, release period, release fraction, cohort geometry, emergency/routine split, non-fielding delay, local residual-pressure evidence, and declared rate scenario.
  arXiv  Detail & Related papers  (2026-07-05T21:23:26Z)
• Verify-Gated Completion as Admission Control in a Governed Multi-Agent Runtime: A Bounded Architecture Case Study [0.6875312133832079]
  We study verify-gated completion as an admission-control pattern for governed multi-agent runtimes.<n>We show that under observed conditions, a read-only verify gate plus packetized admission records made completion decisions inspectable and fail-closed.
  arXiv  Detail & Related papers  (2026-05-18T07:52:13Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• SBEST: Spectrum-Based Fault Localization Without Fault-Triggering Tests [17.90798133817018]
  This study investigates the feasibility of using stack traces from crash reports as proxies for fault-triggering tests in Spectrum-Based Fault localization.<n>We propose SBEST, a novel approach that integrates stack trace information with test coverage data to perform fault localization when fault-triggering tests are missing.
  arXiv  Detail & Related papers  (2024-05-01T15:15:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.