A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis

🕒 Published (v1): 2025-11-24 13:23 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KDR-Agent is a multi-agent LLM framework for low-resource, multi-domain named entity recognition (NER) that replaces retrieval-dependent demonstrations with static contrastive examples and augments inference with Wikipedia knowledge retrieval, entity disambiguation, and reflective self-correction. It outperforms both zero-shot and few-shot in-context learning (ICL) baselines across ten benchmark NER datasets and three LLM backbones. The framework is accepted at AAAI 2026.

Problem

Existing ICL-based NER methods face three compounding limitations in low-resource, multi-domain settings: (1) few-shot ICL requires large annotated corpora for effective demonstration retrieval, which is unavailable in low-resource conditions; (2) zero-shot ICL relies on LLM internal knowledge that fails on novel or specialized domains; and (3) neither paradigm addresses the need for external domain knowledge or systematic resolution of ambiguous entity mentions (e.g., "Apple" as ORG vs. LOC).

Method

KDR-Agent operates in two stages:

Stage 1 — Knowledge In-context Construction: A Central LLM Planner scans the input to identify domain-specific concepts and ambiguous mentions. It dispatches targeted Wikipedia queries to a Knowledge Retrieval Agent (\(k_i = \text{RetrieveFromWiki}(q_i)\)) and flags ambiguous mentions for a Disambiguation Agent (\(P_\text{disamb} = \text{Disambiguate}(x, P_\text{ambig})\)). The enriched prompt combines six components: task instruction \(P_\text{task}\), type definitions \(P_\text{type}\), static contrastive demonstrations \(P_\text{demo}\), retrieved knowledge \(P_\text{know}\), disambiguation cues \(P_\text{disamb}\), and the input \(x\). Static demonstrations use entity-level positive–negative contrastive pairs: for each gold mention–type pair \((m_p, t_p)^+\), one perturbed negative \((m_n, t_n)^-\) is constructed by applying one of four error types (span boundary shift, wrong type, spurious mention, omission).

Stage 2 — Reflection & Correction: A Reflective Analysis Agent examines the initial prediction \(\hat{y}^{(0)}\) against four predefined error criteria (span error, type error, spurious detection, omission) and produces a structured diagnostic report \(R = \text{Reflect}(x, \hat{y}^{(0)}, P_\text{Reflection})\). A second inference pass yields the final prediction: $\(\hat{y}^{(1)} = \text{LLM}(x, \hat{y}^{(0)}, P_\text{Reflection}, R, P_\text{Correction})\)$

Key Contributions

• KDR-Agent: a four-agent framework (Central Planner, Knowledge Retrieval Agent, Disambiguation Agent, Reflective Analysis Agent) for multi-domain low-resource ICL NER
• Static entity-level positive–negative contrastive demonstrations that eliminate dependency on large annotated retrieval corpora
• Two-stage pipeline: enriched prompt construction via external knowledge + structured self-correction via reflective analysis
• Evaluation across 10 datasets, 5 domains, 3 LLM backbones (GPT-4o, DeepSeek-V3, Qwen-2.5-72B)

Results

• KDR-Agent consistently outperforms all zero-shot (ZS) and few-shot (FS) ICL baselines across all 10 datasets and all 3 backbones
• Selected GPT-4o F1 scores vs. best baseline (CodeIE):
• BC5CDR: 82.47 vs. 77.61; NCBI: 79.41 vs. 71.97
• MIT Movie: 76.16 vs. 70.67; MIT Restaurant: 69.98 vs. 56.93
• CoNLL-2003: 83.34 vs. 83.01; OntoNotes 5.0: 71.85 vs. 65.67
• Twitter Broad: 74.90 vs. 69.69; Twitter NER-7: 60.87 vs. 52.20
• WikiANN: 74.37 vs. 63.76; WNUT-17: 80.78 vs. 69.91
• Ablation (GPT-4o, selected datasets): removing Reflection drops NCBI from 79.41→75.91; removing the Disambiguation Agent drops Twitter NER-7 from 60.87→55.81; removing Knowledge Retrieval Agent and Disambiguation Agent together drops NCBI to 74.16

Limitations

• Wikipedia retrieval is limited to introductory summaries; deep or cross-lingual domain knowledge is not captured
• Central Planner is capped at 5 Wikipedia queries and 5 disambiguation targets per input, which may be insufficient for highly complex or entity-dense sentences
• The static contrastive demonstration set requires some manual annotation effort upfront, even if modest
• Computational cost is higher than single-pass ICL due to multiple agent calls per input (planner, retrieval, disambiguation, reflection)
• Evaluation does not include cost/latency analysis vs. retrieval-based baselines

Relevance to Harnesses / Meta-Harnesses

KDR-Agent is a concrete instantiation of a task-specific multi-agent harness: a Central LLM Planner acts as a meta-controller that decomposes the prediction task, selects which specialized agents to invoke, and routes intermediate outputs between them in a fixed two-stage pipeline. The reflective correction stage is a canonical example of a self-repair loop — an agent critiques its own output against predefined criteria and re-prompts for correction — a pattern directly applicable to harness design for quality control. The static demonstration strategy and structured prompt assembly (combining type definitions, contrastive examples, external knowledge, and disambiguation in a principled schema) offer a reusable template for prompt-construction harnesses in low-resource NLP settings. This work shows measurable gains from orchestration overhead, providing empirical grounding for multi-agent harness design decisions.