Skip to content

AVDA: Autonomous Vibe Detection Authoring for Cybersecurity

🕒 Published (v1): 2026-03-26 21:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AVDA is a detection-engineering automation framework that uses the Model Context Protocol (MCP) to expose organizational context (existing detections, telemetry schemas, style guides) to LLMs generating executable cybersecurity detection code. Three authoring strategies—Baseline (zero-shot), Sequential (deterministic RAG), and Agentic (ReAct loop)—are benchmarked across 92 production detections, 11 models, and 21 configurations. Agentic workflows achieve 19% quality gain over Baseline; Sequential reaches 87% of Agentic quality at 40× lower token cost.

Problem

Detection authoring (translating threat descriptions into executable query/analytics code) remains largely manual despite detection-as-code maturity. Enterprise portfolios fragment across heterogeneous platforms (KQL, PySpark, Scala), causing duplication, inconsistency, and slow coverage growth. Existing LLM approaches lack grounded access to organizational context—live telemetry schemas and prior detections—and none rigorously compare workflow automation levels (zero-shot vs. RAG vs. agentic) at production scale.

Method

AVDA builds a four-layer harness:

  1. Detection Data Processing: Ingests 10,127 detection artifacts; applies platform-specific parsers with LLM canonicalization to produce a unified schema (id, platform, language, MITRE mappings, data sources, raw code). Outputs are stored in a dual store: vector DB (OpenAI text-embedding-3-large) for semantic search and SQLite for structured metadata.

  2. MCP Server: Exposes 11 tools across three categories—Retrieval (semantic_search, search_by_mitre, search_by_platform, get_content), Schema (get_telemetry_fields, get_supported_actions, get_actions_and_tables, get_best_table), and Similarity (get_similar, get_details).

  3. Authoring Workflows (the core experimental variable):

  4. Baseline: Single zero-shot LLM call with only the detection specification.
  5. Sequential: Deterministic 4-step RAG pipeline—extract metadata → semantic_search (top-10) → get_content (top-3) → generate with exemplars.

  6. Agentic: ReAct loop where the LLM autonomously reasons and invokes any MCP tool in any order, up to 150 iterations.

  7. Evaluation Metric: Composite of four signals—LLM-as-a-judge \(S_{\text{LLM}}\) (GPT-4.1, 10 binary criteria, Eq. 1), embedding cosine similarity \(S_{\text{embed}}\) (Eq. 2), ROUGE-L \(S_{\text{ROUGE-L}}\) (Eq. 3), and normalized Levenshtein \(S_{\text{Lev}}\) (Eq. 4). Validated against practitioner ratings on a stratified 22-detection subset.

Key Contributions

  • AVDA framework: MCP-grounded RAG harness for detection code generation embedded in VS Code and CI/CD pipelines.
  • Empirical comparison: 11 models Ă— 21 configurations Ă— 92 production detections Ă— 3 workflow strategies → 5,796 generated artifacts; quantified quality/cost/latency trade-offs.
  • Validated evaluation methodology: 10-criterion LLM-as-a-judge decomposition validated with statistically significant expert correlation (\(\rho = 0.64\), \(p < 0.002\)).
  • Diagnostic failure-mode analysis: Identifies TTP matching (99.4% pass) and syntax validity (95.9%) as strengths; exclusion logic parity (8.9%) as the key failure mode.
  • Practitioner guidance: Reasoning-capable models outperform; Sequential is the cost-effective production choice; schema hallucination and missing exclusion logic are primary defect classes.

Results

  • Agentic mean overall similarity score: 0.447 (+19% over Baseline).
  • Sequential achieves 87% of Agentic quality at 40Ă— lower token cost.
  • TTP matching pass rate: 99.4% (Q1).
  • Syntax validity pass rate: 95.9% (Q4).
  • Exclusion parity pass rate: 8.9% (Q6) — worst-performing criterion by large margin.
  • Expert vs. automated metric correlation: Spearman \(\rho = 0.64\), \(p < 0.002\) on 22-detection stratified subset.
  • Corpus covers 92 detections, 5 platforms, 3 languages, 14 MITRE tactics, 66 techniques; complexity ranges from 12 to 3,280 LoC (median 94).

Limitations

  • Evaluation corpus is internal to Microsoft; external generalizability is unconfirmed.
  • Only 2 of 5 platforms expose schema via MCP tools, limiting agentic schema-grounding to a subset of detections.
  • Exclusion logic (the worst-performing criterion) requires organizational operational history that the retrieval corpus cannot always provide.
  • Expert validation subset is small (22 detections); broader human evaluation is not reported.
  • Publicly available Sentinel/XDR documentation may have contaminated model pretraining, making zero-shot baselines optimistic for those platforms.
  • The paper is truncated; full model leaderboard and per-platform breakdowns (RQ5–RQ8) are not shown in the provided text.

Relevance to Harnesses / Meta-Harnesses

AVDA is a concrete domain-specific harness that orchestrates LLM calls through a structured MCP tool layer—precisely the pattern that meta-harness research aims to generalize. Its three-tier workflow taxonomy (zero-shot → deterministic RAG pipeline → autonomous ReAct loop) mirrors the orchestration spectrum studied in agentic harness literature, and its empirical finding that a fixed sequential pipeline captures 87% of agentic quality at 40× lower cost is directly relevant to the cost-quality trade-off question central to harness design. The MCP-as-context-bus architecture demonstrates one production instantiation of tool-augmented code generation harnesses, while the validated composite evaluation metric (LLM-judge + embedding + syntactic) offers a reusable harness evaluation methodology transferable to other code-generation domains.