Skip to content

TACTIC-KG: Toward Small Agent Teams for Cyber Threat Intelligence Knowledge Graph Construction

🕒 Published (v1): 2026-07-06 12:41 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TACTIC-KG is a multi-agent agentic framework that constructs Cybersecurity Knowledge Graphs (CSKGs) from unstructured Cyber Threat Intelligence (CTI) reports by decomposing the task into four specialized agents—Extractor, Typer, Verifier, and Curator—each fine-tuned via LoRA on 3B–8B parameter models. It targets the core failure modes of monolithic LLM pipelines: hallucination, over-completion, and ontology drift. On human-annotated CTI benchmarks, agent specialization outperforms larger monolithic in-context-learning (ICL) baselines in extraction F1, typing accuracy, and graph structural similarity.

Problem

Monolithic LLM approaches to CSKG construction (e.g., CTINEXUS, IntelEX) suffer from: (1) hallucinated triples that propagate unchecked through the pipeline; (2) over-completion—artificially adding edges to force graph connectivity beyond what the text supports; (3) poor performance of smaller models under large-context, joint extraction+typing regimes; and (4) high inference cost from requiring frontier models. Traditional NLP pipelines (NER, rule-based) are brittle to linguistic variation and cannot capture implicit adversarial relations.

Method

TACTIC-KG decomposes CSKG construction into a sequential four-agent pipeline, each agent LoRA-fine-tuned on a lightweight base model (3B–8B):

  1. Extractor Agent: Generates raw candidate triples \((h, r, t)\) from semantically chunked CTI text (sentence-boundary segmentation with overlap). Computes token-level log-likelihood confidence scores per triple. No typing is performed at this stage.

  2. Typer Agent: Given extracted triples, assigns ontology-compliant semantic types \(\tau_h, \tau_t \in \mathcal{O}\), mapping \((h, r, t) \to (h, \tau_h, r, t, \tau_t)\). Restricted to a closed label space; assigns Uncategorized when evidence is insufficient to prevent ontology drift. Fine-tuned with the ontology schema embedded directly in the prompt.

  3. Verifier Agent: Binary classifier \(\mathcal{V}(h, \tau_h, r, t, \tau_t, x) \to \{\text{SUPPORTED}, \text{NOT\_SUPPORTED}\}\) operating under a closed-world assumption on the source text. Supports explicit and implicit entailment. Extracts a minimal evidence span \(e \subset x\) for HITL traceability. Trained with auto-generated negatives from non-supported or weakly-implied relations.

  4. Curator Agent: Document-level sanitization: deduplication, entity normalization (alias merging, case normalization), ontology compliance enforcement, and constrained structural repair. May introduce new bridging triplets only when grounded in text; trained under a minimality objective on cross-component latent relations.

Pipeline: CTI Report → Semantic Chunking → Extractor → Typer → Verifier → Curator → Curated CSKG

Intermediate outputs are serialized as JSON; no agent modifies upstream spans; partial re-execution is supported. LoRA adapters are injected into attention projection layers (\(q\_\text{proj}\), \(v\_\text{proj}\)) independently per agent.

Key Contributions

  • A four-stage agentic pipeline with the design principle faithfulness before connectivity, enforcing textual grounding as a first-class constraint at every stage.
  • Task decomposition that reduces per-agent "task entropy," enabling 3B–8B LoRA-adapted models to outperform large frontier ICL baselines.
  • A generate-then-verify paradigm via a dedicated Verifier Agent, isolating hallucination at the triplet level before graph assembly.
  • A Curator Agent with a minimality objective for constrained structural repair, preventing spurious graph completion.
  • Explicit handling of over-completion as a distinct failure mode from hallucination, with structural metrics to quantify it.
  • HITL-friendly protocol: serialized JSON intermediates, no upstream modification, auditable evidence spans.
  • Comparative ontology coverage analysis (MALOnt, ATT&CK, STIX 2.1) motivating ontology-grounded specialization.

Results

The paper is truncated before the numerical results tables; only the following quantitative and qualitative claims appear in the text:

  • TACTIC-KG (Ministral-3B + LoRA) extracts and types all entities in the FORCEDENTRY/Pegasus CTI example; CTINEXUS (DeepSeek-V3.1-671B + ICL) misses attacker roles, temporal information, and portions of target typing in the same report (Fig. 1).
  • Abstract claims: agent specialization "consistently outperforms larger monolithic ICL baselines in extraction F1-score, typing accuracy, and structural graph similarity" on human-annotated CTI benchmarks (CTI-HAL + CTINEXUS benchmark corpora).
  • CTINEXUS "exhibits significantly degraded CSKG quality when using smaller models," whereas TACTIC-KG is designed for small-model compatibility.
  • Full quantitative tables (RQ1–RQ5) are not present in the provided text due to truncation.

Limitations

  • Evaluation datasets (CTI-HAL, CTINEXUS benchmark) may not cover the full linguistic diversity of real-world CTI reports across all vendor styles.
  • LoRA fine-tuning requires labeled CTI corpora; negative samples for the Verifier are synthetically generated and may introduce supervision bias.
  • Sequential pipeline design means latency accumulates across four agent stages; no parallelization strategy is described.
  • The curator's constrained repair relies on textual evidence grounding, but implicit relation inference (permitted for implicit support) introduces residual hallucination risk.
  • Semantic chunking with overlap may produce duplicate triple candidates; merging heuristics are not fully detailed.
  • Quantitative results are unavailable in the provided text, limiting independent assessment of claimed gains.

Relevance to Harnesses / Meta-Harnesses

TACTIC-KG is a canonical example of a task-decomposition harness: it replaces a single monolithic model call with an orchestrated pipeline of specialized agents, each constrained to a narrow sub-task. The design principles—modular agents, serialized intermediate state, support for partial re-execution, and HITL insertion points—directly instantiate harness architecture patterns. The framework demonstrates that small LoRA-adapted models within a well-structured harness can outperform large frontier models invoked as a single-pass oracle, which is a key empirical argument for investing in harness design over model scaling. The explicit error-containment strategy (no agent modifies upstream outputs, hallucinations are pruned locally before propagation) is a harness-level correctness guarantee rather than a model-level one, making this a strong case study for harness-first engineering in structured information extraction.