Skip to content

Knowledge-Centric Agents for Workflow Generation

🕒 Published (v1): 2026-07-17 11:03 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; knowledge-centric agents for workflow generation in visual creation systems (ComfyUI)

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a knowledge-centric framework for automatically generating ComfyUI workflows (JSON-encoded DAGs for visual AI pipelines) by inverting hierarchical knowledge from real workflows and injecting it into LLMs via SFT. Rather than treating workflow generation as direct text-to-JSON, the system learns explicit strategy, structure, and reasoning representations that enable agentic, multi-level reasoning from task instruction to executable workflow.

Problem

Existing LLM-based workflow generation approaches treat ComfyUI workflow construction as direct text-to-JSON generation, suffering from structural brittleness (misconnected nodes, invalid parameter bindings) and lacking the tacit, experiential knowledge required for expert-level module composition. Annotating this expertise at scale is impractical, and models trained end-to-end entangle knowledge opaquely in weights, making it hard to interpret or reuse across abstraction levels.

Method

The framework operates in three stages:

Knowledge Inversion — Starting from a curated collection of 912 real ComfyUI workflows, the system constructs a 5-level hierarchy bottom-up: (1) Full Pseudo-code: rule-based regularization removing ComfyUI-specific syntax while preserving topology and parameters; (2) Skeleton Pseudo-code: LLM-driven removal of parameter bindings, retaining only graph structure; (3) Strategy Reasoning: LLM inference of decision traces (why branches are split, why face-detailing follows denoising, etc.); (4) High-level Strategy: concise verbal description of the processing plan; (5) Task Instruction: inferred natural-language user intent.

Knowledge Injection — SFT on Qwen3-14B (LoRA, rank 16, \(\alpha=32\)) for two learned transitions: Task→Strategy and Strategy→Skeleton Pseudo-code. The Skeleton→Full Pseudo-code step is handled rule-based. An auxiliary objective of strategy reasoning extraction regularizes the model toward interpretable expert logic.

Knowledge Inference — At test time, inference reverses inversion: a High-Level Strategy Planner generates a strategy, a Pseudo-code Generator produces skeleton pseudo-code, a pretrained LLM (GPT-5) fills parameters, and rule-based reconstruction yields the final executable JSON workflow. Self-refinement loops after strategy and pseudo-code generation improve structural coherence.

Key Contributions

  • A knowledge inversion paradigm that recovers explicit, hierarchical representations (pseudo-code, skeleton, strategy, reasoning traces, task instruction) from unannotated real-world workflows without manual annotation.
  • A three-phase framework (invert → inject → infer) that separates knowledge acquisition from model training and makes knowledge reusable across abstraction levels.
  • Demonstration that SFT is the primary capability carrier for agentic workflow reasoning, while RL-style alignment (GRPO) is secondary; supported by ablation comparison with ComfyUI-R1.
  • A curated dataset of 912 high-quality ComfyUI workflows with three evaluation splits (high/medium similarity and real-world out-of-domain cases), plus evaluation on FlowBench (30 tasks) and ComfyBench (20 tasks).

Results

Metrics: Valid Node Diversity (VND), Node Completeness (NodeComp), Link Completeness (LinkComp), Task Consistency (TaskCons). Baselines: Zero-Shot, Few-Shot, CoT, CoT-SC, RAG, ComfyAgent — all using GPT-5 as the underlying LLM.

  • The proposed method outperforms all six baselines on richer node diversity (VND), more coherent connectivity (LinkComp), and higher task consistency (TaskCons) across all three in-domain splits (Challenge 1, Challenge 2, Real-world Cases).
  • Achieves higher execution success rates than existing systems on both in-domain and out-of-domain benchmarks (FlowBench 30 tasks; ComfyBench Creative + Complex 20 tasks).
  • (Full numerical tables are referenced but not reproduced in the provided text excerpt.)

Limitations

  • Evaluation set is small (30 in-domain test workflows; 30 FlowBench; 20 ComfyBench tasks), limiting statistical robustness.
  • Training corpus of 912 workflows is narrow relative to the full diversity of community-created ComfyUI workflows; the authors note generalization to emerging community workflows remains a challenge.
  • Strategy inversion and reasoning extraction steps rely on large LLMs (GPT-4o, GPT-5), making the inversion pipeline itself expensive and API-dependent.
  • Parameter filling at the Skeleton→Full Pseudo-code stage uses a pretrained LLM without fine-tuning, potentially being a weak link for precise numerical configurations.
  • Self-refinement adds inference-time cost; no latency or token-budget analysis is provided.

Relevance to Agentic AI / LLM Agents

This work is a concrete case study in knowledge-grounded agentic planning: the agent does not generate outputs end-to-end but instead reasons through an explicit hierarchy of representations, mirroring how human experts decompose complex tasks. The knowledge inversion idea — extracting structured, reusable reasoning traces from unlabeled demonstrations — is directly applicable to other agentic domains where tacit expertise is hard to annotate (e.g., multi-step code agents, robotic task planning). The finding that SFT dominates RL for injecting genuinely new compositional capabilities (rather than preference alignment) is an important empirical data point for the agent training community. The self-refinement loop and reversible reasoning design also contribute to the growing literature on iterative agent self-correction.