Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems

• URL: http://arxiv.org/abs/2605.06457v1
• Date: Thu, 07 May 2026 15:50:26 GMT
• Title: Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems
• Abstract summary: Agentic Success Rate (ASR) is a trajectory-fidelity metric that compares observed and expected agent execution sequences.<n>ASR is applied to the Hierarchical Multi-Agent System for Payments (HMASP) across 18 LLMs and 90,000 task instances.<n>Prompt refinements and deterministic routing guards guided by ASR diagnostics yield substantial TSR improvements.
• Score: 1.8717456484053328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based multi-agent systems are increasingly deployed for payment workflows, yet prevailing metrics, Task Success Rate (TSR) and Agent Handoff F1-Score (HF1), capture only final outcomes or unordered routing decisions. We introduce the Agentic Success Rate (ASR), a trajectory-fidelity metric that compares observed and expected agent execution sequences at the transition level, decomposing performance into Transition Recall and Transition Precision. Applied to the Hierarchical Multi-Agent System for Payments (HMASP) across 18 LLMs and 90,000 task instances, ASR reveals that 10 of 18 models systematically skip a confirmation checkpoint during payment checkout, a deviation invisible to both TSR and HF1, while 8 models enforce the checkpoint perfectly. Notably, GPT-4.1 exhibits hidden workflow shortcuts despite achieving perfect TSR and HF1, while GPT-5.2 achieves perfect ASR. Prompt refinements and deterministic routing guards guided by ASR diagnostics yield substantial TSR improvements, with gains up to +93.8 percentage points for previously struggling models, demonstrating that trajectory-level evaluation is essential in regulated domains.

Related papers

• TACIT-Switch: Cost-Aware Model Escalation for LLM Agents from Censored Supervision [2.681967953218225]
  Agents with smaller language-model backbones can drift into persistent failure modes, whereas those with larger backbones are generally more reliable but more costly.<n>This reliability-cost trade-off motivates routing methods that decide when to invoke an agent with a larger backbone.<n>Our method, TACIT-SWITCH, learns permanent handoff policies from accumulated trajectory evidence and Teacher-Annotated Censored Intervention Times.
  arXiv  Detail & Related papers  (2026-08-28T04:33:11Z)
• TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents [22.01777615106231]
  Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes.<n>We propose Transition-wise Credit Assignment, which derives step-level supervision directly from action-induced transitions.<n>Experiments on ALFWorld, WebShop, and seven search-augmented question-answering benchmarks show consistent improvements over the evaluated baselines.
  arXiv  Detail & Related papers  (2026-08-17T06:19:12Z)
• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution [6.908637308550535]
  SEVA is a structured verification agent that emits evidence alignments, step-by-step reasoning chains, calibrated confidence, and a six-category error diagnosis.<n>On ClearFacts, SEVA-3B matches GPT-4o-mini (69.0 vs. 69.8 F1) while producing substantially richer, auditable output.
  arXiv  Detail & Related papers  (2026-06-29T02:37:13Z)
• TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agents [62.969415789328025]
  LLM agents can pursue hidden malicious objectives through sequences of individually benign actions.<n>Existing approaches either evaluate complete trajectories in a single pass or partition them into independently scored windows.<n>We propose TRACE, a monitoring framework for long-horizon LLM agent trajectories.
  arXiv  Detail & Related papers  (2026-06-05T08:54:38Z)
• AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking [13.891522069967507]
  We present AgentEval, a framework that formalizes agent executions as evaluation directed acyclic graphs (DAGs)<n>AgentEval achieves 2.17x higher failure detection recall than end-to-end evaluation and 72% root cause accuracy against an 81% human ceiling.
  arXiv  Detail & Related papers  (2026-04-26T07:38:47Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• LLM Readiness Harness: Evaluation, Observability, and CI Gates for LLM/RAG Applications [51.56484100374058]
  We present a readiness harness for LLM and RAG applications that turns evaluation into a deployment decision workflow.<n>The system combines automated benchmarks, OpenTelemetry observability, and CI quality gates under a minimal API contract.<n>We evaluate the harness on ticket-routing and BEIR tasks with full Azure matrix coverage.
  arXiv  Detail & Related papers  (2026-03-28T18:03:32Z)
• AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling [12.42698406379544]
  AgentHER recovers lost training signal by adapting the Hindsight Experience principle to natural-language agent trajectories.<n>It converts discarded failures into high-quality SFT, DPO, and ShareGPT training data.<n>On WebArena Replay (Zhou et al., 2024) and ToolBench (Qin et al., 2024), AgentHER improves over success-only SFT by +7.1-11.7 pp across four model families.
  arXiv  Detail & Related papers  (2026-03-22T18:36:58Z)
• The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents [1.3299507495084417]
  We study how runtime enforcement against unsafe actions affects end-to-end task performance in multi-step tool using large language model (LLM) agents.<n>We identify model dependent interaction horizons (15 to 30 turns) and decompose outcomes into overall success rate (SR), safe success rate (SSR), and unsafe success rate (USR)
  arXiv  Detail & Related papers  (2026-03-18T16:16:34Z)
• Verified Critical Step Optimization for LLM Agents [67.05296684575445]
  Critical Step Optimization focuses preference learning on verified critical steps.<n>Method starts from failed policy trajectories rather than expert demonstrations.<n>Experiments on GAIA-Text-103 and XBench-DeepSearch show that CSO achieves 37% and 26% relative improvement over the SFT baseline.
  arXiv  Detail & Related papers  (2026-02-03T11:41:02Z)
• ProRe: A Proactive Reward System for GUI Agents via Reasoner-Actor Collaboration [24.005812534024965]
  ProRe is a proactive reward system that leverages a general-purpose reasoner and domain-specific evaluator agents.<n>ProRe improves reward accuracy and F1 score by up to 5.3% and 19.4%, respectively.
  arXiv  Detail & Related papers  (2025-09-26T03:29:36Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Exploring Expert Failures Improves LLM Agent Tuning [74.0772570556016]
  We propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories.<n>EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance.
  arXiv  Detail & Related papers  (2025-04-17T17:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.