Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents

🕒 Published (v1): 2026-07-16 05:08 UTC · Source: Arxiv · link

Why this paper was selected

Industry coding-agent benchmark (Alipay) covering realistic repo-level payment integration tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Alipay-PIBench is a repository-level benchmark for evaluating AI coding agents on realistic Alipay payment integration tasks, spanning 9 payment products and 18 task instances across two progressive scenarios: functional payment completion and risk-aware hardening. A paired without-skill/with-skill protocol measures how structured domain guidance (the alipay-payment-integration skill) affects agent performance. Across six evaluated models, mean Rubric Pass Rate (RPR) under the with-skill condition ranges from 68.58% to 91.37%, with an average +10.31 pp gain over no-skill.

Problem

Existing coding benchmarks (HumanEval, SWE-bench, LiveCodeBench) measure general code correctness but are not designed to assess domain-specific correctness requirements such as payment-product selection, server-side credential handling, asynchronous notification verification, idempotent state transitions, and payment–business state consistency. A functionally runnable payment integration can still introduce real fund-safety risks or business-state inconsistencies that no generic test catches.

Method

Product-Grounded Task Construction: Nine Alipay payment products (In-App, JSAPI, Mobile Web, PC Web, QR Code, Order QR Code, Authorization Hold, Merchant Initiated Transaction, Usage Based) are each paired with a real open-source business repository (diverse stacks: Node.js/Express, Laravel/PHP, Java/Spring Boot, Next.js/TypeScript). Each product–project pair yields two independently initialized task instances: - Basic (Functional Payment Completion): agent starts from a repository with no target-product integration and must implement the full end-to-end payment loop. - Advanced (Risk-Aware Payment Hardening): agent starts from a repository where the core integration already exists and must harden it against idempotency failures, duplicate notifications, abnormal transaction states, refund risks, and fund-safety boundary violations.

Rubric-Derived Evaluation: Each scenario has a scenario-specific rubric decomposing the task into checkable requirements. Rubric items are instantiated as (a) deterministic evaluators—static checks (credential placement, SDK usage, signature-verification hooks), unit tests, integration tests (using Alipay sandbox or deterministic mock fixtures for non-sandbox-reproducible states), and E2E tests—and (b) supplementary LLM-assisted criteria for semantic properties (product fit, state-machine coherence, cross-component linkage, security-sensitive logic placement). Both channels derive from the same shared rubric, keeping evidence traceable to domain requirements.

Paired Skill Intervention: Every task is run twice with the same agent, project, and evaluators: once without the alipay-payment-integration skill (baseline) and once with it (structured guidance including product selection, SDK/sandbox guidance, code examples, and common pitfalls). The primary metric is RPR (Rubric Pass Rate), reported at method level to distinguish source-level completion, executable payment behavior, and payment-domain requirements.

Key Contributions

• First Alipay-led repository-level benchmark for evaluating coding agents on realistic Alipay payment integration (18 task instances × 9 products × 2 scenarios).
• Product-Grounded Task Construction framework unifying payment products, business repositories, scenario states, and natural-language integration instructions.
• Progressive capability decomposition into Basic and Advanced scenarios, separating end-to-end payment completion from risk hardening.
• Rubric-Derived Evaluation methodology: deterministic checks and LLM-assisted criteria co-derived from the same scenario-specific rubric contract.
• Paired without-skill/with-skill protocol to measure the practical effect of structured domain guidance on payment safety, product selection, state consistency, and output-token efficiency.

Results

• With-skill mean RPR across six models: 68.58%–91.37%.
• With-skill improves over without-skill by +10.31 pp on average.
• Gains are positive in 101 of 108 model–product–scenario comparison pairs.
• Basic tasks show larger average skill gain (+11.27 pp) than Advanced tasks (+9.35 pp).
• Method-level results differentiate source-level completion, executable payment behavior, and payment-domain rubric adherence (exact per-model numbers not fully reported in the provided excerpt).

Limitations

• Only 18 task instances across 9 products; coverage of the full Alipay Open Platform product surface is partial.
• Benchmark is Alipay-ecosystem-specific; generalization to other payment platforms (Stripe, PayPal, WeChat Pay) is not evaluated.
• LLM-assisted rubric verification is acknowledged to be noisy in agentic coding contexts (per RuVerBench findings cited in the paper).
• Some Advanced scenarios involving rare abnormal states or sandbox-unsupported products rely on deterministic mock fixtures rather than live sandbox behavior, which may not capture all real-world edge cases.
• The alipay-payment-integration skill is a proprietary artifact; external reproducibility of the with-skill condition depends on access to it.
• Only six models evaluated; the benchmark's discriminative power across a broader model population is unknown from the provided text.

Relevance to Agentic AI / LLM Agents

Alipay-PIBench is a direct contribution to the growing line of work pushing coding agent evaluation beyond syntactic correctness toward domain-specific, safety-critical, multi-component task completion—a key challenge for deploying LLM agents in production software workflows. The paired skill-intervention protocol is particularly relevant: it operationalizes how structured tool/skill injection (analogous to RAG, MCP tools, or agent-steering prompts) changes agent behavior on security- and consistency-sensitive tasks, providing a principled measurement methodology that could transfer to other high-stakes domains. The progressive Basic→Advanced scenario decomposition offers a reusable design pattern for benchmarks that need to separate capability tiers (can the agent complete the task vs. can it harden it) within a single evaluation framework.