Skip to content

Characterizing Large Language Model Agentic Workflows: A Study on N8n Ecosystem

🕒 Published (v1): 2026-06-27 23:57 UTC · Source: Arxiv · link

Why this paper was selected

Empirical characterization of real-world LLM agent workflows; valuable for harness designers

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents the first large-scale empirical study of LLM agentic workflows in the n8n low-code automation ecosystem, analyzing 6,003 publicly available workflow templates. It finds that LLMs are primarily used as text generation (31.1%), information extraction (18.3%), and planning/agentic execution (16.6%) components embedded in non-trivial workflow topologies, yet explicit reliability and governance mechanisms remain rare.

Problem

LLM agent benchmarks overwhelmingly focus on isolated capabilities (QA, code generation, reasoning) evaluated in controlled settings. There is no systematic empirical characterization of how LLM agents are actually deployed and composed within real-world workflow automation platforms—where agents are embedded in graphs with control logic, external tools, storage, communication services, and human review gates. This gap matters because agent behavior in practice is shaped by workflow structure, not just model capability.

Method

The authors collect 6,003 public n8n workflow JSON files containing LLM-related components (OpenAI, Anthropic, Hugging Face, Gemini, Mistral, Ollama, LangChain, AI Agent, chat model nodes), filtered to exclude disconnected templates and near-duplicates. They parse each workflow into a normalized graph with two complementary views: a main execution graph (83,201 nodes, 83,305 edges) capturing control flow, and an AI dependency graph (21,228 AI auxiliary nodes, 20,728 edges) modeling configuration links between AI root nodes and components (language models, tools, memory, output parsers, retrievers). Workflow primary tasks are annotated using an LLM (DeepSeek-v4-flash), achieving 96% accuracy on a 200-sample manual validation set. Structural analysis covers topology classification, node functional categories (deterministic rule-based), tool-use patterns, reliability mechanisms, and human-control placement via static graph analysis of the workflow JSON.

Key Contributions

  • First large-scale empirical study of LLM-based automation workflows, bridging the gap between benchmark evaluation and real-world deployment
  • Analysis of 6,003 public n8n workflows covering task distribution, structural patterns, tool compositions, failure-handling strategies, and autonomy/governance
  • A two-view graph analysis methodology (main execution graph + AI dependency graph) that separates control flow from AI component configuration
  • Release of dataset, analysis framework, and replication package

Results

  • Task distribution: Text generation dominates (31.1%), followed by information extraction (18.3%), planning/agentic execution (16.6%), classification (9.7%), question answering (8.5%), summarization (7.0%), and data analysis/reporting (6.7%). Code generation, translation, and moderation together account for <2.1%.
  • Topology diversity: Only 23.27% of workflows follow a strict linear chain; 22.32% contain branching/merging, 18.92% contain cyclic structures, and 18.74% have multi-component topologies.
  • LLM-based annotation outperforms keyword-based (40.5%) and zero-shot NLI-based (14.5%) approaches, achieving 96% accuracy on a 200-sample manual validation set.
  • Explicit reliability mechanisms remain uncommon—structured fallback paths, repair loops, failure-specific alerts, and human approval gates are relatively rare despite the prevalence of LLM deployment.
  • 5,022 of 6,003 workflows contain an LLM node directly in the main execution graph; the remaining 981 include LLM components only as AI auxiliary nodes.

Limitations

  • Static analysis of workflow JSON captures design-time intent rather than runtime behavior—the paper explicitly notes that JSON does not reveal whether fallback branches are actually triggered under realistic inputs
  • The study is limited to a single platform (n8n); generalizability to other low-code ecosystems (Zapier, Make, etc.) is unaddressed
  • Task annotation relied on an LLM (DeepSeek-v4-flash) rather than exhaustive manual labeling; the 96% accuracy on 200 samples may not hold uniformly across the full 6,003 dataset
  • Workflow topology categories are applied at the graph level and may conflate structurally similar but semantically different patterns
  • The dataset consists of publicly shared templates, which may differ systematically from private, production-deployed workflows

Relevance to Harnesses / Meta-Harnesses

This paper provides the first empirical ground truth for what "LLM agentic workflow" actually means in practice, revealing a large gap between benchmark-centric evaluations and real-world deployment patterns. For the meta-harness topic, it suggests that harness designers should move beyond single-turn, single-model evaluations and account for workflow-topology factors (branching, cycles, multi-component graphs), explicit reliability mechanisms, and human-in-the-loop control points. The finding that 16.6% of workflows involve planning/agentic execution—but that these are embedded in broader automation structures—indicates that harnesses need to evaluate agent behavior within an execution graph context, not in isolation.