Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents

• URL: http://arxiv.org/abs/2607.14573v3
• Date: Wed, 22 Jul 2026 02:38:57 GMT
• Title: Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents
• Abstract summary: We introduce the Alipay-PIBench, a benchmark for evaluating coding agents on realistic payment integration.<n>It contains nine product-specific projects and 18 task instances, each organized into Basic functional-completion and Advanced risk-aware hardening scenarios.<n>We evaluate six coding-agent models and report rubric pass rate (RPR)<n>Under the with-skill condition, mean RPR ranges from 68.58% to 91.37%.<n>Access to the alipay-payment-integration skill improves mean RPR by 10.31 percentage points on average relative to the without-skill condition.
• Score: 14.209118550299165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Payment integration is a demanding repository-level software task: agents must select a suitable product, implement coordinated client-server flows, verify payment outcomes, and preserve consistency between transaction and business states. We introduce Alipay-PIBench, a benchmark for evaluating coding agents on realistic Alipay payment integration. It contains nine product-specific projects and 18 task instances, each organized into Basic functional-completion and Advanced risk-aware hardening scenarios. Scenario-specific rubrics support deterministic static, unit, integration, and end-to-end checks, supplemented by LLM-assisted assessment for semantic requirements. We evaluate six coding-agent models and report rubric pass rate (RPR). Under the with-skill condition, mean RPR ranges from 68.58% to 91.37%. Access to the alipay-payment-integration skill improves mean RPR by 10.31 percentage points on average relative to the without-skill condition, with gains varying across models, products, and scenarios. Method-level results distinguish source-level completion, executable payment behavior, and payment-domain requirements. Alipay-PIBench provides a controlled setting for diagnosing model capability and evaluating structured guidance in payment integration.

Related papers

• DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents [16.576661790025668]
  DAREBench is a benchmark designed to capture workload variation and support reliable agent evaluation.<n>No single model dominates all workload groups, text and multimodal tasks exhibit distinct accuracy--cost trade-offs.<n>These findings suggest that agent deployment and model selection should consider workload profiles, deployment mode, and accuracy--cost trade-offs.
  arXiv  Detail & Related papers  (2026-09-05T12:36:59Z)
• PACE: Policy-Attested Contract Execution for Safe AI Agents in Decentralized Finance [9.22772901752634]
  We present PACE, a transaction-level authorization framework that interposes between an LLM-based agent and on-chain execution.<n> PACE cryptographically bind the approved intent, policy, and simulation report to the exact execution bytes, with replay and expiration protection.<n> PACE achieves a 0.00 unsafe execution rate and 0.00 false-positive rate on benign tasks, compared to 0.80 for the unguarded baseline.
  arXiv  Detail & Related papers  (2026-08-18T00:22:46Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams [14.784600350594614]
  LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier.<n>Existing agentic benchmarks evaluate fixed models or fixed agent configurations, and offer limited guidance for cost-accuracy-optimal deployment.<n>We introduce AgentCARD, a role-aware benchmark suite for evaluating LLM agent teams across role assignment and deployment mode.
  arXiv  Detail & Related papers  (2026-05-28T17:12:20Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis [5.748806787695848]
  Agent-centric Interface (ADI) is a cost-efficient, end-to-end autonomous interaction tool.<n>ADI resolves 63.8% of the tasks on the SWE-bench benchmark.<n>ADI is a plug-and-play component into existing SOTA agents, delivering consistent gains ranging from 6.2% to 18.5% on resolved tasks.
  arXiv  Detail & Related papers  (2026-04-27T09:18:42Z)
• AVDA: Autonomous Vibe Detection Authoring for Cybersecurity [0.1633272850273525]
  We introduce AVDA, a framework that leverages the Model Context Protocol (MCP) to automate detection by integrating organizational context into AI-assisted code generation.<n>We evaluate three authoring strategies - Baseline, Sequential, and Agentic - across a diverse corpus of production detections and state-of-the-art LLMs.<n>Our results show that Agentic attains 87% of Agentic quality at 40x lower token cost.
  arXiv  Detail & Related papers  (2026-03-26T21:52:33Z)
• $τ$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge [58.03692489021332]
  $$-Knowledge is an extension of $$-Bench for evaluating agents in environments where success depends on coordinating external, natural-language knowledge with tool outputs.<n>We show that $$-Knowledge provides a realistic testbed for developing agents that integrate unstructured knowledge in human-facing deployments.
  arXiv  Detail & Related papers  (2026-03-04T18:34:47Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• On the problem of entity matching and its application in automated settlement of receivables [47.187609203210705]
  We consider setup, where base algorithm is used for preliminary ranking of matches. We apply several novel methods to increase matching quality of base algorithm.
  arXiv  Detail & Related papers  (2022-05-21T21:16:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.