A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis¶

🕒 Published (v1): 2025-11-24 13:23 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

KDR-Agent is a multi-agent LLM framework for low-resource, multi-domain named entity recognition (NER) that replaces dynamic retrieval-based demonstration selection with static contrastive examples, Wikipedia-grounded knowledge retrieval, a disambiguation agent, and a reflective correction stage. It consistently outperforms zero-shot and few-shot ICL baselines across 10 NER benchmarks spanning 5 domains under GPT-4o, Qwen-2.5-72B, and DeepSeek-V3 backbones.

Problem¶

Existing in-context NER methods have three compounding weaknesses in low-resource, multi-domain settings: (1) few-shot ICL requires large annotated support sets for effective retrieval; (2) zero-shot ICL depends on LLM internal domain knowledge, which is insufficient for novel domains like biomedical; (3) both paradigms neglect external knowledge injection and systematic entity disambiguation, causing errors on ambiguous or domain-specific mentions (e.g., "Apple" as ORG vs. LOC).

Method¶

KDR-Agent operates in two stages:

Stage 1 — Knowledge In-context Construction: A central LLM planner receives the input text and entity type set, then (i) generates Wikipedia search queries for domain-specific or rare mentions and (ii) flags ambiguous mentions. A Knowledge Retrieval Agent fetches Wikipedia lead paragraphs for each query. A Disambiguation Agent resolves flagged ambiguities via contextualized self-check dialogues. The final prompt concatenates: task instruction, natural-language type definitions, a static set of entity-level positive–negative contrastive demonstrations (perturbing gold annotations with span errors, type errors, spurious entities, or omissions), retrieved knowledge snippets, and disambiguation explanations.

Stage 2 — Reflection & Correction: The NER module generates an initial prediction ŷ⁽⁰⁾ from the enriched prompt. A Reflective Analysis Agent then audits ŷ⁽⁰⁾ against four predefined error categories (span error, type error, spurious detection, omission), producing a structured diagnostic report R. A second LLM call incorporating R yields the final prediction ŷ⁽¹⁾.

Key Contributions¶

KDR-Agent multi-agent framework integrating knowledge retrieval, disambiguation, and reflective correction for multi-domain low-resource in-context NER.
Entity-level positive–negative contrastive demonstration construction that eliminates dependence on large annotated retrieval pools (5–10 static examples suffice).
Central LLM planner that decomposes task into targeted Wikipedia queries and ambiguity-flagging, coordinating specialized agents.
Structured four-category reflection protocol (span, type, spurious, omission) enabling post-hoc self-correction without fine-tuning.

Results¶

KDR-Agent outperforms all zero-shot and few-shot ICL baselines on all 10 datasets across all 3 LLM backbones.
Largest gains in biomedical (BC5CDR, NCBI) and social media (Twitter Broad, Twitter NER-7) domains.
Representative GPT-4o F1 scores vs. best prior baseline (Code-IE): BC5CDR 82.47 vs. 77.61; NCBI 79.41 vs. 71.97; MIT Restaurant 69.98 vs. 56.93; Twitter NER-7 60.87 vs. 52.20; WNUT-17 80.78 vs. 69.91.
Ablation (GPT-4o): removing reflection drops NCBI from 79.41→75.91, Twitter NER-7 from 60.87→57.81; removing both KRA and DA drops NCBI to 74.16; removing negative contrastive samples drops all datasets modestly.
Backbone scale study (Qwen): NCBI F1 degrades from 79.22 (72B) to 65.31 (7B); Twitter NER-7 from 60.73 (72B) to 32.70 (7B); OntoNotes 5.0 degrades more moderately (67.37→54.64).

Limitations¶

Wikipedia retrieval fails silently when no matching article exists; the framework provides no fallback knowledge source.
Central planner caps Wikipedia queries and disambiguation mentions at 5 per input, potentially missing complex multi-entity sentences.
Static contrastive demonstrations require at least some manually annotated examples; the paper uses 5–10, which may still be unavailable in extreme low-resource scenarios.
Performance degrades substantially with smaller LLMs (≤14B), limiting applicability where large proprietary or 70B+ models are impractical.
Reflection stage adds a full additional LLM inference call per instance, increasing latency and cost.
Evaluation restricted to English; cross-lingual generalization is untested.

Relevance to Agentic AI / LLM Agents¶

KDR-Agent exemplifies the planner–executor multi-agent pattern applied to a structured prediction task: a central LLM planner decomposes the problem, delegates subtasks to specialized agents (retrieval, disambiguation, reflection), and synthesizes their outputs into a final decision. The reflective correction stage is a concrete instantiation of the Reflexion/self-critique paradigm, demonstrating that verbal self-assessment loops measurably reduce structured output errors. For researchers tracking agentic LLM systems, this paper shows how multi-agent coordination with tool use (Wikipedia API) and role specialization can replace the need for large curated datasets, a generalizable principle for low-resource agentic pipelines.