Knowledge-Centric Agents for Workflow Generation¶

🕒 Published (v1): 2026-07-17 11:03 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Knowledge-Centric Agents introduces a framework for automatic ComfyUI workflow generation that treats structured domain knowledge—not raw text-to-JSON mapping—as the primary currency. It inverts hierarchical representations (strategy, pseudo-code, topology) from unannotated real-world workflows, injects them via SFT, and reverses the process at inference to synthesize executable DAG pipelines. The approach outperforms prior methods in node diversity, structural coherence, and execution success rate.

Problem¶

Existing LLM approaches to ComfyUI workflow generation treat the task as direct text-to-JSON synthesis, which fails due to (1) syntactic brittleness of large discrete graphs, (2) combinatorial explosion of valid topologies, and (3) the absence of the tacit, experiential knowledge that human designers rely on to decide module selection, branching logic, and control interactions. Models lack multi-level structured knowledge spanning abstract strategy to concrete parameter binding.

Method¶

The framework operates in two symmetric phases:

Knowledge Inversion (offline, bottom-up): Starting from a curated corpus of 912 real ComfyUI workflows (filtered from 10,000+), a rule-based regularizer converts raw JSON graphs into Full Pseudo-code (topology + parameters, no ComfyUI styling). An LLM then strips parameters to produce Skeleton Pseudo-code, infers Strategy Reasoning (why modules are arranged as they are), summarizes it into a High-Level Strategy, and finally reconstructs the missing Task Instruction for workflows lacking explicit prompts.

Knowledge Injection (training): Two transitions require SFT on Qwen3-14B (LoRA rank 16, \(\alpha=32\), single H200 GPU): - Task → Strategy (sequence length 2048) - Strategy → Skeleton Pseudo-code (sequence length 4096)

The Skeleton → Full Pseudo-code step is handled rule-based; no additional SFT needed.

Knowledge Inference (runtime): Given a user instruction, a trained High-Level Strategy Planner produces a strategy; a trained Pseudo-code Generator converts it to skeleton pseudo-code; a pretrained LLM (GPT-5-turbo) fills parameters; a rule-based reconstructor produces the executable ComfyUI JSON. Self-refinement loops are applied after both the strategy and pseudo-code stages.

Key Contributions¶

Knowledge Inversion paradigm: systematic distillation of hierarchical representations (strategy, reasoning, pseudo-code) from unannotated workflows without manual annotation
Hierarchical SFT pipeline: explicit injection of experiential knowledge at task→strategy and strategy→structure transitions
Reversible inference architecture: inference mirrors the inversion hierarchy, enabling principled step-by-step workflow synthesis
Curated ComfyUI dataset: 912 high-quality workflows with multi-stage deduplication and three evaluation splits (Challenge 1/2 + Real-world Cases), plus external evaluation on FlowBench and ComfyBench subsets
Self-refinement at multiple abstraction levels: structural coherence checks at both strategy and pseudo-code stages

Results¶

(Paper text truncated before quantitative tables; only qualitative claims appear in the provided text.) - Outperforms six baselines (Zero-Shot, Few-Shot, CoT, CoT-SC, RAG, ComfyAgent) across Valid Node Diversity (VND), Node Completeness (NodeComp), Link Completeness (LinkComp), and Task Consistency (TaskCons) - Evaluated in-domain (own 30-workflow test set across three difficulty tiers), on FlowBench (30 tasks), and on ComfyBench Creative+Complex subset (20 tasks) - All baselines and the proposed method use GPT-5-turbo as the LLM interface for fairness

Limitations¶

Test set is small (30 workflows total, 20 tasks on ComfyBench subset); generalization claims rest on limited held-out data
Knowledge Inversion relies on LLMs (GPT-4o) for semantic deduplication and strategy extraction—quality is coupled to the capability of those models
SFT on 882 workflows may underfit for rare or highly compositional task types; the paper notes that when new community workflows emerge, template-based prior methods require retraining (the same concern implicitly applies here)
No human evaluation of generated workflows is reported in the provided text
The 10% structural-difference threshold for deduplication is heuristic and may discard meaningfully distinct workflows

Relevance to Harnesses / Meta-Harnesses¶

This paper is a concrete instantiation of a workflow-generation meta-harness: it constructs the scaffolding that assembles task-specific agent pipelines (ComfyUI DAGs) automatically rather than executing a fixed pipeline. The Knowledge Inversion → Injection → Inference loop is itself a meta-harness pattern—a system that learns how to build harnesses from examples of prior harnesses. The self-refinement loops and hierarchical planner/generator decomposition reflect the same structural concerns (correctness gating, multi-stage orchestration, structural coherence) that appear in general-purpose meta-harness research. For researchers tracking harnesses, this work is notable for showing how domain-specific tacit knowledge can be made explicit and learnable so that a meta-harness can generalize to novel task compositions without hand-authored templates.