Skip to content

Extracting Events Like Code: A Multi-Agent Programming Framework for Zero-Shot Event Extraction

🕒 Published (v1): 2025-11-17 08:17 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent-Event-Coder (AEC) reframes zero-shot event extraction (ZSEE) as a multi-agent code-generation pipeline: four specialized LLM agents (Retrieval, Planning, Coding, Verification) collaborate to produce Python class instantiations that are deterministically validated against event schemas. A dual-loop refinement mechanism patches code based on compiler-style diagnostics and backtracks through ranked hypotheses when repair fails. AEC consistently outperforms all five zero-shot baselines across five domains and six LLMs.

Problem

Zero-shot event extraction with LLMs fails in two ways: (i) contextual ambiguity — polysemous trigger words are misclassified without in-context examples; (ii) structural infidelity — direct prompting produces outputs that violate schema constraints (wrong roles, hallucinated arguments, malformed types). Single-agent prompt strategies cannot reliably enforce both semantic correctness and structural validity simultaneously.

Method

AEC decomposes ZSEE into four agents operating in a fixed pipeline:

  1. Retrieval Agent (\(A_{ret}\)): self-generates \(k\) schema-aligned exemplar sentences via analogical prompting — no external retrieval corpus required.
  2. Planning Agent (\(A_{plan}\)): produces a ranked list of \(k\) trigger–type hypotheses \(\{(z_i, e), \beta_i, \rho_i\}_{i=1}^k\) with confidence scores \(\beta_i \in [0,1]\) and natural-language rationales \(\rho_i\).
  3. Coding Agent (\(A_{code}\)): converts the top-ranked hypothesis into executable Python that instantiates a Pydantic BaseModel compiled from the event schema \(S_e = \langle e, \{(r_j, \tau_j)\}_{j=1}^m \rangle\).
  4. Verification Agent (\(A_{verify}\)): runs three deterministic test cases — T1 semantic check (trigger \(z^* \in T\) and semantically compatible with \(e\)), T2 type check (Pydantic field-type enforcement), T3 structural check (valid compilation, exact field set, serializability). Returns \((V, \varepsilon)\).

Dual-loop refinement (Algorithm 1): inner loop patches code up to \(t\) times using diagnostic \(\varepsilon\); outer loop backtracks to next hypothesis when \(t\) attempts are exhausted. Total search space is \(O(kt)\) candidate paths with guaranteed schema-valid output.

Key Contributions

  • First formulation of ZSEE as a multi-agent code-generation task unifying schema constraints, iterative planning, and code-level validation.
  • Schema-as-code verification: event schemas compiled to Pydantic BaseModel classes, enabling runtime deterministic validation with typed feedback.
  • Dual-loop refinement algorithm combining inner-loop code patching with outer-loop hypothesis backtracking.
  • Comprehensive evaluation across five domains (FewEvent, ACE 2005, GENIA, SPEED, CASIE) and six LLMs (Llama3-8B/70B, Qwen2.5-14B/72B, GPT-3.5-turbo, GPT-4o).

Results

  • On ACE 2005 with Llama3-8B: AEC achieves TI=40.5, TC=48.8, AI=33.7, AC=31.8 — gains of +7.8% TI and +6.0% TC over ChatIE.
  • On ACE 2005 with Llama3-70B: AEC achieves TI=57.0, TC=54.6, AI=38.4, AC=34.7 — best across all baselines.
  • Generalization (Table 2): AEC outperforms DecomposeEE (strongest baseline) by approximately +3–5% TI, +4–6% TC, +2–4% AI/AC averaged across Qwen2.5 and GPT-4o families.
  • Ablation (Llama3-70B, ACE): removing Verification Loop drops TI from 57.0→47.1 (−9.9); removing Retrieval Agent drops TI to 49.8 (−7.2); removing Planning Rationales drops TI to 52.6 (−4.4); removing Structural Check drops TI to 54.9 (−2.1).
  • GPT-4o achieves highest overall performance (TI=44.6/TC=42.8 on FewEvent), followed by Qwen2.5-72B, showing scaling effects within the AEC framework.

Limitations

  • Each extraction call invokes four LLM agents plus up to \(O(kt)\) refinement turns — significantly higher inference cost than single-pass baselines.
  • Retrieval Agent self-generates exemplars rather than retrieving from a corpus; quality of self-generated analogies is unconstrained and unverified.
  • Evaluation uses sampled subsets (250 instances per dataset, 50 for CASIE) rather than full test sets; results may not fully reflect distribution at scale.
  • No fine-tuned models tested; performance ceiling with supervised ZSEE methods is not compared.
  • Backtracking strategy is greedy (next-best \(\beta\)); no global optimization across the hypothesis space.

Relevance to Harnesses / Meta-Harnesses

AEC is a textbook task-decomposition harness: a fixed orchestration layer that routes subtasks to specialized agents, collects structured outputs, and drives iterative refinement loops — the same architectural pattern underlying meta-harnesses that coordinate heterogeneous LLM workers. The dual-loop refinement with compiler-style diagnostics is directly analogous to harness feedback loops that detect failures and trigger re-execution or rollback. The schema-as-code verification component demonstrates how deterministic runtime checks can serve as the harness's correctness oracle, a broadly applicable pattern for any harness that must enforce output contracts. For researchers building meta-harnesses, AEC provides a concrete case study of how to wire retrieval, planning, execution, and verification agents into a pipeline with well-defined inter-agent communication protocols (structured Python objects) and formal stopping conditions.