Towards Agentic Investigation of Security Alerts¶

🕒 Published (v1): 2026-04-28 16:52 UTC · Source: Arxiv · link

Why this paper was selected

Agentic multi-source log correlation automates early-stage security alert investigation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

The paper presents an agentic workflow that chains three LLM components (Investigator, Summary, Verdict) with constrained SQL and grep tooling over Suricata IDS logs to automate early-stage SOC alert investigation. The pipeline significantly outperforms a bare LLM on true/false positive classification. It is evaluated on four models across 100 runs each using the AIT Log Data Set V1.1.

Problem¶

Security Operations Centers suffer from alert fatigue: analysts receive high volumes of low-context IDS alerts, can only triage a fraction, and must manually correlate across heterogeneous log sources. Directly prompting an LLM on raw, high-volume unstructured log data is insufficient due to context limits and hallucination risk.

Method¶

A three-component iterative loop, each component being a fresh LLM call with a distinct role:

Investigator LLM receives the alert text plus an overview query result (event counts, top SIDs/IPs) and selects up to 4 predefined SQL queries (e.g., sids_window, top_src_alerts, http_paths_alerts), one optional freeform SELECT over the Suricata DuckDB table, and a mandatory grep on unstructured auth/syslog files.
Execution step runs the selected queries with strict guardrails: all SQL is parameterized, time-bounded (ts BETWEEN ? AND ?), hard-limited to rows in $[1, 5]$, and freeform SQL is validated to be a single SELECT with no mutating keywords. Artifacts (syntax_ok, rows, ms) are logged.
Summary LLM consumes execution results and produces a behavior-focused executive summary, IoCs, MITRE ATT&CK technique tags with confidence scores in $[0,1]$, and an auth_signal.
Incident Verdict LLM issues one of four verdicts: Malicious, Benign, Uncertain, or Requires Further Investigation (triggers a second loop iteration; max 2 iterations currently enforced).

Models tested: GPT-5-mini, claude-3-haiku-20240307, qwen3:30b (local), gemma3:27b (local). Each model runs 100 complete workflow executions against both a malicious subset (active attack phase) and a benign subset (quiet window) from AIT Log Data Set V1.1, with a synthetic low-context alert ("Suspicious behaviour") as the trigger.

Key Contributions¶

A modular, multi-LLM agentic pipeline for alert triage with role-separated components (Investigator / Summary / Verdict).
A constrained tool-access layer: parameterized SQL over DuckDB + regex-validated grep, preventing prompt-injection-driven mutation.
An iterative loop with an explicit "requires further investigation" exit condition capped at 2 iterations with a stated need for a proper loop variant/termination proof.
Comparative evaluation across 4 diverse LLMs (2 API, 2 local) for stochastic robustness (RQ3) at low cost (<$10/100 runs for API models).
Public prompt/query corpus on GitHub (Rub3cula/CyberHunt2025).

Results¶

The agentic workflow achieves significantly higher accuracy in true/false positive classification than a baseline of the same LLM prompted without the workflow (quantitative breakdown was in sections not fully provided in the excerpt).
All four models could successfully operate the workflow (select queries, interpret results, issue verdicts) in a zero-shot setting.
API cost for 100 full runs per model: <$10 USD for GPT-5-mini and claude-3-haiku; local models (qwen3:30b, gemma3:27b) complete 100 runs in <1 hour on an RTX 4090.

Limitations¶

Evaluation uses a single synthetic alert ("Suspicious behaviour") as the trigger; generalization to diverse real-world alert types is untested.
Maximum loop depth is hard-coded at 2 iterations; no formal termination proof or loop variant is provided.
Dataset is small and sourced from a single academic dataset (AIT Log Data Set V1.1); real SOC data scarcity is acknowledged.
Grep over unstructured logs is rudimentary; no semantic or embedding-based retrieval.
Verdict categories (especially Uncertain vs. Requires Further Investigation) are coarsely defined; confidence sub-tiers for benign cases are acknowledged as missing.
The paper is truncated before full quantitative results and discussion sections are presented.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a concrete instantiation of a domain-specific LLM harness: a fixed-topology pipeline of specialized LLM calls wired together with constrained tool execution, artifact logging, and a conditional loop with a guarded retry count. The guardrail layer (parameterized SQL, regex validation, row-count caps, mutation keyword blocklist) is a disciplined harness-side enforcement pattern directly analogous to tool-access sandboxing in general meta-harness designs. The iterative loop with an explicit termination condition and the modular swap-out of models/queries/log-sources reflects the same configurability concerns that harness frameworks address. For someone tracking harnesses, this paper demonstrates how to operationalize role-separated LLM chains, structured artifact passing between stages, and safety constraints on agentic tool use within a narrow but high-stakes domain.