KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KARMA is a nine-agent LLM pipeline that automates knowledge graph (KG) enrichment from scientific literature by decomposing the task into ingestion, reading, summarization, entity extraction, relation extraction, schema alignment, conflict resolution, and evaluation stages. Applied to 1,200 PubMed articles across genomics, proteomics, and metabolomics, it discovers up to 38,230 new entities with 83.1% LLM-verified correctness and reduces conflict edges by 18.6% versus a single-agent baseline.

Problem

Manual KG curation cannot keep pace with >7 million scientific articles published annually. Single-agent LLM approaches suffer from hallucination, schema inconsistency across documents, and quadratic compute costs on full-text inputs. No existing system integrates entity discovery, schema alignment, and conflict resolution in a cohesive, verifiable pipeline.

Method

KARMA decomposes KG enrichment into nine specialized LLM agents coordinated by a Central Controller: 1. Ingestion Agents normalize raw PDFs and extract metadata. 2. Reader Agents segment text and score section relevance against the existing KG (discard if R(s_j) < δ). 3. Summarizer Agents condense segments to reduce noise for downstream agents. 4. Entity Extraction Agents run LLM-based NER filtered by domain ontology, then normalize entities via embedding-space nearest-neighbor matching; entities exceeding distance threshold ρ are flagged as new candidates. 5. Relationship Extraction Agents classify entity-pair relations via probability thresholding (p ≥ θ_r), supporting multi-label prediction. 6. Schema Alignment Agents map new entities/relations to existing KG types via LLM classification over the valid type set T. 7. Conflict Resolution Agents run an LLM-based debate prompt to detect logical incompatibilities with existing edges; contradicting triples are discarded or queued for expert review. 8. Evaluator Agents compute per-triple confidence, clarity, and relevance scores (sigmoid-weighted aggregation); triples with mean score < Θ are rejected.

Backbone LLMs evaluated: GLM-4 (9B), GPT-4o, and DeepSeek-v3 (37B MoE).

Key Contributions

• Nine-agent modular architecture covering the full KG enrichment pipeline with cross-agent verification.
• Domain-adaptive prompting strategies enabling extraction across heterogeneous scientific subfields without retraining.
• LLM-based debate mechanism for conflict resolution that reduces contradictory edge insertion.
• Multi-faceted evaluation framework (core metrics, graph statistics, quality indicators including human evaluation) for KG assessment without a gold-standard reference dataset.
• Ablation study isolating contributions of Summarizer, Conflict Resolution, and Evaluator agents.

Results

• Coverage gain: DeepSeek-v3 discovers 38,230 new entities from 720 genomics papers (53.1/article), vs. 4,384 for single-agent baseline (3.6× improvement on a per-article basis in genomics vs. metabolomics).
• LLM-verified correctness (R_LC): 83.1% (DeepSeek-v3, genomics); GPT-4o peaks at 88.0% (genomics) but yields 41% lower connectivity gain than DeepSeek-v3.
• Conflict reduction: 18.6% of extracted edges removed by Conflict Resolution Agent (genomics, DeepSeek-v3); conflict ratios range 0.132–0.238 across domains and models.
• Human evaluation (R_HE): KARMA-Full (DeepSeek-v3) scores 0.625 in genomics vs. 0.320 for single-agent baseline.
• QA coherence (C_QA): 0.612 (DeepSeek-v3, genomics) vs. 0.493 for single-agent.
• Ablation: Removing Summarizer drops R_LC by 18.2% in proteomics; removing Evaluator drops R_LC by 9.7% in metabolomics; removing Conflict Resolution drops C_QA by 4.9% in genomics.
• DeepSeek-v3 wins on 17/24 metrics across domains.

Limitations

• Evaluation relies primarily on LLM-based correctness scores rather than gold-standard human annotation at scale; human evaluation uses only two experts.
• Tested exclusively on biomedical PubMed literature; generalizability to other domains (finance, law, etc.) is unverified.
• Multi-agent pipeline incurs substantial token and processing costs (completion token mean 550.64, std 232.92 for genomics); no cost comparison against single-agent per-triple is provided.
• Schema Alignment assumes a predefined valid type set T; truly novel entity types still require manual intervention.
• All agents within one experiment share the same LLM backbone; heterogeneous backbone assignment (e.g., GPT-4o for precision-critical stages) is not explored.
• No ground-truth KG diff to measure recall; coverage gain metrics could reward hallucinated but plausible entities.

Relevance to Agentic AI / LLM Agents

KARMA is a concrete instantiation of task decomposition via specialist agents — a core pattern in agentic LLM research — applied to a domain (KG construction) where correctness is verifiable and failure modes (hallucination, schema drift) are well-defined. The LLM-based debate mechanism in the Conflict Resolution Agent is directly related to multi-agent debate as an alignment/verification technique (cf. AI safety via debate). The ablation results quantify what each agent role contributes, providing empirical grounding for the widely held intuition that modular, cross-verifying agent pipelines outperform monolithic LLM calls on structured extraction tasks. This work also highlights a recurring challenge in agentic systems: evaluation without ground truth, solved here by layering LLM judges, graph metrics, and human spot-checks.