MAFA: A Multi-Agent Framework for Enterprise-Scale Annotation with Configurable Task Adaptation

🕒 Published (v1): 2025-10-16 00:30 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAFA is a production-deployed multi-agent annotation system from JP Morgan Chase that routes millions of customer utterances through a pipeline of specialized ranker agents and a judge-based consensus mechanism. It eliminates annotation backlogs at enterprise scale via a configuration-driven architecture that adapts to any annotation task (intents, FAQs, entities) without code changes. Deployed at JPMorgan Chase, it cleared a 1M-utterance backlog and saves 5,000+ annotation hours annually.

Problem

Enterprise annotation workflows cannot scale: at JPMorgan Chase, 30,000+ daily utterances outpaced human capacity by 10×, producing a 6-month, 1M-utterance backlog. Single-model LLM annotators lack reliability for production financial services (hallucination, inconsistency, domain specificity). Inter-annotator agreement on the human baseline was only 72%, and annotation cycles delayed model improvements by 6–8 weeks.

Method

MAFA is a four-stage hierarchical pipeline:

1. Query Planning Agent (GPT-4o): Expands ambiguous utterances via two-stage intent analysis + contextual expansion (e.g., "cash back" → "cash back policies, rewards, redemption, credit cards"). Preserves raw input for inherently ambiguous strings. A 24-hour TTL cache reduces API calls by 35%.

2. Four Parallel Ranker Agents (ThreadPoolExecutor, 50 workers): Two prompt-only agents (primary-field-only and full-context) and two embedding-augmented agents using OpenAI text-embedding-3-large (3,072-dim) with Matryoshka Representation Learning (MRL) for adaptive dimensionality. Each agent receives 8–15 unique few-shot examples to maximize ensemble diversity. Parallel execution reduces latency from 2,800 ms (sequential) to 650 ms. Each agent outputs structured JSON (ARQ format) with relevance scores and confidence.

3. Judge Agent (GPT-4o, temp 0.3): Synthesizes all candidate annotations via weighted voting (weights updated daily on rolling accuracy), contextual re-ranking for domain business rules, and audit trail generation. Falls back to simple score aggregation on 200 ms timeout.

4. Configuration-Driven Dispatch: An AnnotationConfig class parameterizes annotation type, column mappings, and match semantics—all agent prompts and output formats auto-adapt from config alone.

Confidence-based routing: 85% HIGH (auto-accept), 10% MEDIUM (auto-accept + flag), 5% LOW (human queue).

Key Contributions

• Production multi-agent annotation harness processing millions of utterances with 86% human-annotator agreement, deployed at JPMorgan Chase.
• Configuration-driven framework allowing task adaptation (intents, FAQs, entities, custom) via config file, zero code changes.
• Few-shot diversity distribution: each ranker receives unique examples to maximize ensemble coverage.
• Empirical ablation quantifying each component's contribution (judge: −5.7%, embeddings: −4.2%, query planning: −2.5%, few-shot diversity: −2.8% when removed).
• Batch API path for non-real-time workloads at 50% cost reduction; per-query cost of $0.003 (85% reduction vs. single-agent).

Results

All comparisons are 1-agent vs. Full MAFA (mean ± std over 10 runs with random few-shot seeds):

• Internal Banking (500K queries, 150 intents): Top-1 0.699→0.837 (+13.8%), Top-5 0.776→0.927 (+15.1%), F1 0.741→0.910 (+16.9%); latency 13.5→20.5 s (batch mode).
• Banking77 (13K queries, 77 intents): Top-1 0.768→0.873 (+10.5%), Top-5 0.841→0.968 (+12.7%), F1 0.798→0.945 (+14.7%).
• CLINIC-150 (23.7K queries, 150 intents, 10 domains): Top-1 0.838→0.901 (+6.3%), Top-5 0.930→0.980 (+5.0%), F1 0.900→0.970 (+7.0%).
• Full MAFA is statistically significant vs. 4-Agent baseline (\(p < 0.01\), paired t-test) on all datasets.
• Judge agent alone contributes the largest single-component gain (5.7% on Internal Banking).
• System sustains 1,000 QPS with burst to 2,500 QPS; linear scalability to 8 Kubernetes instances.

Limitations

• Latency cost of full MAFA is substantially higher than simpler baselines (20.5 s vs. 13.5 s on Internal Banking); the accuracy–latency tradeoff may be prohibitive for synchronous, low-latency applications.
• All experiments use GPT-4o (OpenAI); no evaluation with open-weight or on-premise models, limiting applicability where data privacy prevents external API calls (despite stated PII masking).
• Embedding-based agents rely on pre-computed indices; novel or rapidly changing annotation taxonomies require index rebuilds.
• Few-shot example selection uses stratified sampling but the pool construction and selection strategy's sensitivity is not fully characterized.
• Evaluation is on intent classification and FAQ mapping only; generalization to other annotation types (entity extraction, sentiment) claimed but not benchmarked.
• Scalability degrades beyond 8 instances due to coordination overhead; root cause not analyzed.

Relevance to Harnesses / Meta-Harnesses

MAFA is a textbook production meta-harness: it wraps multiple heterogeneous sub-agents (prompt-only, embedding-augmented, query planner, judge) behind a configurable dispatch layer that abstracts task semantics entirely into configuration rather than code. The AnnotationConfig pattern—where a single config object parameterizes all agent prompts, routing logic, and output schemas—is directly analogous to the task-description interfaces found in meta-harness design. The judge-plus-consensus mechanism implements a meta-level synthesis layer over first-pass agents, mirroring the reviewer/aggregator pattern common in harness architectures. For researchers tracking harnesses, MAFA provides a rare production-scale case study with quantified ablations on each orchestration layer (planner, ensemble diversity, judge), concrete latency/cost tradeoffs, and a confidence-routing design that tightly integrates human-in-the-loop override into the harness control flow.