Skip to content

Extracting Events Like Code: A Multi-Agent Programming Framework for Zero-Shot Event Extraction

🕒 Published (v1): 2025-11-17 08:17 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent-Event-Coder (AEC) reframes zero-shot event extraction (ZSEE) as a multi-agent code-generation pipeline, where event schemas are expressed as executable Python classes and four specialized agents (Retrieval, Planning, Coding, Verification) collaboratively produce schema-compliant event objects. A dual-loop refinement mechanism patches generated code using compiler-style diagnostics and backtracks to lower-confidence trigger hypotheses when exhausted. AEC consistently outperforms all zero-shot baselines across five domains and six LLMs without any labeled training data.

Problem

Zero-shot event extraction with LLMs suffers from two failure modes: (1) contextual ambiguity — polysemous trigger words are misclassified without training examples to anchor interpretation; (2) structural infidelity — direct prompting produces outputs that violate predefined event schemas (missing arguments, hallucinated roles, wrong data types). Single-agent prompting approaches lack the iterative error-correction needed for complex structured prediction.

Method

AEC decomposes ZSEE into a four-agent pipeline: 1. Retrieval Agent: self-generates k exemplar sentences for the target event schema using analogical prompting, grounding subsequent agents in concrete linguistic realizations. 2. Planning Agent: examines input text with exemplars, outputs a ranked list of k trigger–type hypotheses each with a confidence score β and natural-language rationale ρ. 3. Coding Agent: converts the highest-confidence hypothesis into executable Python that instantiates a Pydantic BaseModel encoding the event schema; schema compliance reduces to constructing a valid class instance. 4. Verification Agent: executes a three-stage test suite — Semantic Check (trigger appears in text and is semantically compatible), Type Check (argument values match declared types, enforced via Pydantic), Structural Check (code compiles, fields match schema exactly). Returns (V, ε) where ε is the first failing check.

A dual-loop refinement procedure runs up to t patch attempts per hypothesis (inner loop), then backtracks to the next-ranked hypothesis (outer loop), exploring O(kt) candidate paths deterministically.

Key Contributions

  • First framework to reformulate ZSEE as a multi-agent code-generation task, unifying schema constraints and iterative verification.
  • Schema-as-code representation: event schemas as executable Python BaseModel classes, enabling deterministic runtime validation rather than heuristic output checking.
  • Dual-loop refinement: inner patch loop driven by compiler-style diagnostics; outer loop backtracks through confidence-ranked trigger hypotheses.
  • Comprehensive evaluation across five domain-diverse benchmarks (FewEvent, ACE 2005, GENIA, SPEED, CASIE) and six LLMs (Llama3-8B/70B, Qwen2.5-14B/72B, GPT-3.5-turbo, GPT-4o).

Results

  • ACE 2005 (Llama3-8B): AEC achieves TI=40.5, TC=48.8, AI=33.7, AC=31.8 — gains of +7.8% TI and +6.0% TC over best baseline (ChatIE).
  • ACE 2005 (Llama3-70B): AEC achieves TI=57.0, TC=54.6, AI=38.4, AC=34.7, surpassing all baselines.
  • GPT-4o generalization (ACE): AEC reaches TC=61.8, AC=36.8, vs. DecomposeEE TC=58.4, AC=35.3.
  • Average gains across four LLMs (Table 2): approximately +3–5% TI, +4–6% TC, +2–4% in argument metrics over strongest baseline DecomposeEE.
  • Ablation (Llama3-70B, ACE): removing Verification Loop drops TC from 54.6 to 44.7; removing Retrieval Agent drops TI from 57.0 to 49.8; removing Planning Rationales drops TI to 52.6.
  • Performance saturates at k=3 hypotheses, t=3 patch attempts, and 3 verification test cases.

Limitations

  • Increased inference cost: O(kt) agent calls per extraction instance vs. single-pass baselines; computational overhead grows with k and t.
  • Performance gains saturate and can slightly decline beyond k=3, t=3, suggesting fragility to over-refinement with noisy low-confidence hypotheses.
  • Self-generated exemplars from the Retrieval Agent may introduce factual inconsistencies or domain-inappropriate patterns without ground-truth retrieval.
  • Evaluation uses 250 uniformly sampled test instances per dataset, which may not reflect true dataset distributions; CASIE results use only 50 samples.
  • No analysis of failure modes for highly ambiguous schemas or extremely long documents (GENIA/CASIE average 250+ tokens).

Relevance to Agentic AI / LLM Agents

AEC is a concrete instantiation of the decompose-and-verify multi-agent pattern: specialized agents handle distinct subtasks, and a dedicated verifier enforces correctness via executable checks rather than LLM self-evaluation, directly addressing the unreliability of single-agent structured generation. The schema-as-code principle — converting task constraints into runnable artifacts that generate deterministic feedback — is broadly transferable to other structured extraction and planning tasks where schema fidelity is critical. The dual-loop refinement mirrors the plan-act-reflect cycle common in agentic frameworks, but grounds backtracking in programmatic diagnostics rather than soft language-model judgment. For researchers tracking agentic AI, this paper demonstrates how role specialization and executable verification can together achieve reliable zero-shot performance on tasks where naive prompting consistently fails.