Skip to content

PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contracts

🕒 Published (v1): 2025-11-04 18:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PoCo is an agentic framework that generates executable Foundry proof-of-concept exploit tests for Solidity smart contract vulnerabilities from natural-language auditor annotations. It wraps a single LLM in a ReAct loop with domain-specific compilation and execution tools, all containerized for safety. On 23 real-world vulnerabilities, it produces 3× more logically correct PoCs than a fixed-workflow baseline.

Problem

Manually writing PoC exploits during smart contract audits is slow and error-prone under tight schedules. Prior automated approaches output raw transaction sequences (not executable tests), restrict scope to direct-extraction exploits, or use a fixed pipeline that cannot adapt to diverse real-world project structures. No tool addressed full, executable PoC generation from high-level textual vulnerability descriptions.

Method

PoCo is a single-LLM agent operating in a ReAct (Reason–Act–Observe) loop. The agent receives three inputs: the vulnerable contract path, an auditor-written vulnerability annotation, and a target output path. It autonomously selects and sequences calls to a minimal toolset:

  • Basic tools: glob, grep, read, edit/write for filesystem navigation.
  • Planning tool: a lightweight Todo read/update interface for internal task tracking.
  • Smart-contract tools: smart-contract-compile (Solidity compilation via Foundry Forge) and smart-contract-test (PoC execution against the vulnerable codebase), both running inside Docker containers.

The system prompt encodes domain best practices: vulnerability analysis, Foundry-specific idioms (vm.prank), iterative refinement on compilation/test/logical errors, and minimality constraints (no modification of original contracts). Tool interfaces are exposed as MCP servers via FastMCP; the scaffold is implemented on top of the Claude Code SDK (v2.0.10). Following the "bitter lesson" principle, scaffolding is kept minimal so the LLM controls all tool selection and ordering.

Correctness is evaluated using a novel oracle: the developer-provided mitigation patch. A PoC is logically correct if it passes against the unpatched contract but fails (or the vulnerability assertion triggers) against the patched version. Three models are evaluated: Claude Sonnet 4.5, OpenAI o3, GLM 4.6.

Key Contributions

  • PoCo: an agentic framework for executable PoC exploit generation from natural-language vulnerability descriptions.
  • PRoof-of-Patch dataset: 23 real-world Solidity vulnerabilities with linked source code, auditor annotations, and verified mitigation patches; fully reproducible and open-source.
  • Patch-as-oracle evaluation methodology: uses developer mitigation patches as independent correctness oracles for automated PoC validation, avoiding the need for manual ground-truth PoCs.
  • Empirical campaign: 23 vulnerabilities × 3 models showing PoCo generates 3× more logically correct PoCs than workflow baselines.
  • Agent behavior monitoring: full interaction trajectories (prompts, tool calls, errors) plus execution metadata (tokens, cost, duration, round counts) for post-hoc analysis.

Results

  • PoCo generates 50 well-formed (compilable) PoC exploits across runs (RQ1).
  • PoCo generates 32 logically correct PoC exploits across the 23-vulnerability dataset (RQ2).
  • PoCo achieves the highest success rates on both well-formedness and logical correctness by a large margin over Zero-shot Prompting and Workflow Prompting baselines.
  • Workflow baseline produces approximately one-third as many correct PoCs as PoCo.
  • Performance degrades as auditor annotation detail decreases (RQ3), confirming input sensitivity as a failure mode.

Limitations

  • Dataset is small (23 vulnerabilities); generalization to broader vulnerability classes is unverified.
  • PoCo requires an auditor-written vulnerability annotation — it does not discover vulnerabilities autonomously.
  • Performance is sensitive to annotation quality: sparse descriptions increase the burden on the model to infer implicit semantic and operational constraints.
  • Evaluation relies on patch availability; vulnerabilities lacking verified patches cannot be assessed for correctness.
  • The paper text is truncated before the full results and threats-to-validity sections; additional stated limitations may exist.

Relevance to Harnesses / Meta-Harnesses

PoCo is a concrete, production-oriented harness instantiation: it wraps an LLM with a curated tool scaffold (MCP servers, Dockerized execution, CLI allowlist), a structured system prompt encoding domain heuristics, and an agent behavior monitoring layer that captures full interaction trajectories and resource metadata — a pattern directly analogous to meta-harness observability infrastructure. The comparison of agentic vs. workflow vs. zero-shot baselines provides empirical evidence that minimal-scaffolding, full-autonomy harnesses outperform rigid pipeline harnesses on tasks requiring adaptive multi-step reasoning. The implementation via Claude Code SDK and MCP demonstrates a reusable harness composition pattern (LLM + MCP tool servers + container isolation) that generalizes beyond smart contracts.