Skip to content

ComfyUI-Copilot: An Intelligent Assistant for Automated Workflow Development

🕒 Published (v1): 2025-06-05 13:20 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ComfyUI-Copilot is a hierarchical multi-agent LLM plugin that automates ComfyUI workflow development via intent-driven retrieval, from-scratch code generation, and node/model recommendation. It wraps a central assistant agent coordinating specialized worker agents backed by curated knowledge bases of 7K nodes, 62K models, and 9K workflows. Offline recall exceeds 88.5% and real-world workflow acceptance reaches 85.9%.

Problem

ComfyUI's node-graph paradigm imposes high expertise barriers: sparse documentation, complex dependency management, and the need for deep component knowledge to construct or debug workflows. Prior automated approaches (ComfyGen, ComfyGI) are unstable (generating unparseable JSON) and limited to text-to-image tasks, leaving conditional image/video generation unaddressed.

Method

A LangChain-implemented hierarchical multi-agent framework with one central assistant agent and three specialized worker agents (workflow generation, node recommendation, model recommendation):

  1. Knowledge Base construction: 7K nodes (auto-documented via sandbox execution + BGE-M3 code embedding + LLM synthesis), 62K models, 9K workflows (GPT-4o-enhanced from community sources). Updated weekly.
  2. Three-stage retrieval pipeline: (a) LLM intent expansion, (b) hybrid retrieval scoring \(\text{sim}_O = 0.7 \times \text{sim}_S + 0.3 \times \text{sim}_L\) over OpenAI text-embedding-3-small semantic scores and lexical overlap, yielding top-30 candidates, (c) GTE-Rerank re-ranking to top-3, further sorted by popularity signals.
  3. Workflow generation: workflows are represented as Python-like code converted from JSON via graph topology traversal; top-tier closed-source LLMs (DeepSeek-V3/GPT-4o) generate from scratch given retrieved node exemplars. A fine-tuned Qwen2.5-Coder-7B achieves comparable pass rate to Claude 3.7 Sonnet.
  4. Assistant agent uses short-term memory (message history) to route queries directly or delegate to worker agents, then synthesizes a unified response.

Key Contributions

  • First open-source ComfyUI assistant plugin for automated workflow creation and instant module suggestions.
  • Curated, auto-expanding KB covering 7K nodes, 62K models, 9K workflows with LLM-generated documentation.
  • Hybrid coarse-to-fine retrieval pipeline combining semantic embeddings, lexical overlap, and learning-to-rank reranking.
  • Python-code as intermediate workflow representation enabling LLM-native generation and JSON round-tripping.
  • Fine-tuned open-source model (Qwen2.5-Coder-7B) reaching parity with frontier models on workflow generation pass rate.
  • Auxiliary features: prompt refinement, parallel parameter search, multilingual QA, missing-node installation guidance.

Results

  • Recall@3 (offline, 130 workflow instructions): DeepSeek-V3 90.0%, GPT-4o 89.2%.
  • Recall@3 (offline, 104 node instructions): DeepSeek-V3 88.5%, GPT-4o 89.4%.
  • Online node acceptance rate: 65.4% (19K users).
  • Online workflow acceptance rate: 85.9%.
  • Deployment: 85K queries, 19K users, 22 countries, 1.6K GitHub stars as of May 2025.
  • Fine-tuned Qwen2.5-Coder-7B achieves pass rate and node selection comparable to Claude 3.7 Sonnet on workflow generation (exact numbers not reported beyond "comparable").

Limitations

  • Workflow generation pass rates still have "significant room for improvement" despite high recall; the paper does not report absolute pass-rate numbers for from-scratch generation.
  • Node acceptance rate (65.4%) is substantially lower than workflow acceptance (85.9%), indicating recommendation quality for individual nodes lags.
  • Evaluation is on self-constructed test sets derived from the same KB used for retrieval, risking circularity.
  • No ablation isolating the contribution of each retrieval stage or the reranker independently.
  • Closed-source LLM dependency for highest-quality generation; open-source alternative is "comparable" but exact delta unreported.

Relevance to Harnesses / Meta-Harnesses

ComfyUI-Copilot is a domain-specific meta-harness: it orchestrates a hierarchy of LLM agents over a curated, auto-maintained knowledge base to compose and deploy multi-step AI pipelines on behalf of users. Its architecture — central planner routing to specialized workers, KB-backed retrieval, and code-as-intermediate-representation for structured pipeline generation — directly instantiates the meta-harness pattern where the harness manages both workflow discovery (retrieval) and workflow synthesis (generation). The auto-documentation pipeline for KB construction (sandbox execution → code chunking → LLM synthesis) is itself a sub-harness, relevant to anyone designing self-maintaining harness registries. The tension between retrieval-based and generative workflow construction, and the use of a fine-tuned small model as a drop-in replacement for frontier models, are directly applicable design questions for general-purpose harness builders.