Skip to content

PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements

🕒 Published (v1): 2025-05-31 15:38 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PAKTON is an open-source tri-agent framework for legal contract QA that chains three specialized ReAct agents—Archivist, Interrogator, and Researcher—around a multi-stage hybrid RAG pipeline. It achieves state-of-the-art accuracy on ContractNLI while substantially narrowing performance gaps across backbone LLMs. Its plug-and-play design allows any open-source model to serve as the agent backbone without retraining.

Problem

Contract review requires rare legal expertise, is expensive, and involves confidential documents that cannot be sent to proprietary APIs. Existing LLM approaches lack grounded, explainable retrieval over long documents with cross-referencing clauses, and perform inconsistently across backbone models, making open-source deployment unreliable.

Method

PAKTON orchestrates three agents, each implemented as a ReAct reasoning loop over a pluggable backbone LLM:

  1. Archivist: Parses contracts into a hierarchical tree (using indentation, numbering, BERT/LLM semantic proximity), then generates three complementary chunk types per tree node—node-level, ancestor-aware, descendant-aware—enriched with structural metadata. Embeddings are stored for retrieval.

  2. Researcher: On demand from the Interrogator, autonomously selects among in-document (BM25 + dense embeddings + Reciprocal Rank Fusion, augmented by LightRAG for entity/relation matching), cross-document, and external (web/Wikipedia/SQL/MCP) retrieval. Retrieved passages are reranked by a cross-encoder, sigmoid-normalized, and threshold-filtered.

  3. Interrogator: Iteratively decomposes the user query into sub-questions, dispatches each to the Researcher, drafts a structured report after the first response, identifies knowledge gaps, and issues follow-up questions. Terminates when confidence is sufficient or a user-configured turn limit is reached. Output report includes title, legal reasoning, key findings, knowledge gaps, and cited spans.

Key Contributions

  • Hierarchical tree parsing of legal documents with three structurally-aware chunk types (node-level, ancestor-aware, descendant-aware)
  • Plug-and-play tri-agent architecture with backbone-LLM independence and MCP-based external retrieval
  • Demonstration that the harness compresses inter-model variability: coefficient of variation drops from \(CV > 16\%\) (zero-shot) to \(CV = 5.7\) (PAKTON)
  • Regression slope of \(0.33\) between baseline ZS performance and PAKTON F1[W], meaning each unit of raw model quality gain yields only \(\sim\frac{1}{3}\) unit of PAKTON gain—showing architectural dominance over model choice
  • Qualitative human evaluation by five expert attorneys and a Supreme Court Justice across nine criteria

Results

  • ContractNLI accuracy: PAKTON with Gemma 3 27B achieves 0.8287 vs. Gemma ZS 0.7886; Mistral 7B + PAKTON (0.7032) outperforms standalone GPT-4o few-shot (0.6640)
  • PAKTON vs. domain-specific baseline: Saul 54B (fine-tuned legal LM) achieves 0.7020 accuracy; Mistral 7B + PAKTON matches it at 0.7032
  • Robustness: ANOVA on high- vs. low-ZS model groups under PAKTON yields \(F=4.19, p=0.075\) (no statistically significant difference), confirming architecture-level normalization
  • Qualitative: PAKTON outperforms GPT-4o in 8 of 9 human-evaluated criteria (interpretability, completeness, grounded justifications, etc.)
  • LegalBench-RAG: Results table is truncated in the provided text; PAKTON claimed to achieve state-of-the-art retrieval across NDA, M&A, commercial, and privacy policy domains vs. Naive, RCTS, Naive+Cohere, RCTS+Cohere baselines

Limitations

  • LegalBench-RAG external retrieval tools (web, Wikipedia, SQL, MCP) were disabled during experiments for fairness, so real-world external-knowledge performance is unvalidated
  • Traditional ablation at the agent level is not conducted because the three agents are described as interdependent; component-level contributions remain incompletely isolated
  • Evaluation is primarily on English-language contracts; multilingual or cross-jurisdictional generalization is not tested
  • Iterative interrogation turn depth is a user-set hyperparameter; sensitivity analysis across turn counts is not reported
  • Human evaluation sample size (five attorneys + one judge) is small and may not generalize across legal domains

Relevance to Harnesses / Meta-Harnesses

PAKTON is a concrete domain harness that instantiates the meta-harness pattern: a fixed orchestration layer (Interrogator as dispatcher) wraps swappable backbone LLMs inside typed agent roles, demonstrating that the harness architecture itself—not model quality—drives the majority of performance. The plug-and-play design maps directly to the modular, model-agnostic harness philosophy, with each agent's ReAct loop functioning as a sub-harness with its own tool set. The statistical finding that PAKTON compresses baseline model variance by more than \(2\times\) provides empirical evidence for a general claim relevant to harness designers: well-structured orchestration can substitute for raw model capability, a key justification for meta-harness investment. The MCP integration for external retrieval tools further illustrates how harnesses can compose heterogeneous capability providers under a unified control flow.