COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows¶

🕒 Published (v1): 2026-07-02 05:00 UTC · Source: Arxiv · link

Why this paper was selected

Self-evolving skill harnesses for image workflows; directly relevant to agent memory and reusable skills

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

COMFYCLAW is an agentic framework that controls ComfyUI image-generation workflows through typed graph editing, a VLM verifier that translates visual failures into actionable repair suggestions, and a self-evolving skill library distilled from execution traces. Across four benchmarks, three agent models, and two image backbones, it consistently outperforms harness-only baselines, with evolved skills accounting for roughly 50% of skill invocations in later runs.

Problem¶

Existing workflow-generation agents for tools like ComfyUI treat each prompt independently: refinements that succeed in one run are never promoted to validated, reusable procedures. Static or passively-stored skill libraries repeatedly rediscover the same repairs and cannot accumulate structured procedural knowledge from execution failures and verifier feedback.

Method¶

COMFYCLAW casts workflow construction as a skill-augmented MDP over a directed graph $G = (V, E)$ representing a ComfyUI pipeline. The agent edits the graph via typed actions $a_t \in \mathcal{A}$ (node insertion/removal, edge connection, parameter updates, LoRA insertion, regional conditioning): $$G_{t+1} = \text{Edit}(G_t, a_t).$$ A VLM verifier decomposes the prompt $p$ into a binary requirement checklist $Q = \{q_i\}$, returns per-requirement pass/fail labels plus a holistic detail score $s_\text{det} \in [0,10]$, and emits localized repair suggestions. The harness combines these into a scalar reward: $$r = 0.6 \cdot \frac{|Q_\text{pass}|}{|Q|} + 0.4 \cdot \frac{s_\text{det}}{10}.$$ The refinement loop iterates until $r \geq \tau_\text{stop}$ or $K$ iterations elapse. After every batch of $B$ prompts, execution traces $\mathcal{D}^{(m)} = \{(p_i, G_i, x_i, r_i, f_i, e_i, a_{i,1:T_i})\}$ are clustered by verifier feedback, runtime errors, and workflow actions into success ($r \geq 0.9$) and failure groups. An LLM skill evolver proposes mutations $\mu_\ell \in \{\text{create, revise, reinforce, merge, delete}\}$ to produce candidate libraries $\tilde{S}_\ell^{(m+1)}$. Each candidate is validated on three held-out prompts synthesized per cluster; the mutation is committed only if $\Delta_\ell \geq 0$ (no degradation vs.\ $S^{(m)}$). Skills are stored as SKILL.md files with progressive disclosure: the agent sees lightweight metadata and retrieves full instructions on demand.

Key Contributions¶

A typed, stage-gated graph-editing harness for ComfyUI that exposes construction, enhancement, and refinement as distinct action sets and auto-reverts invalid edits.
A region-level VLM verifier that decomposes prompts into binary requirement checklists and returns localized, workflow-term repair suggestions.
A skill-evolution loop with held-out mutation validation (create/revise/reinforce/merge/delete) that converts recurring trajectories into committed, reusable SKILL.md procedures.
Empirical demonstration across 4 benchmarks × 3 agents × 2 image backbones that skill evolution provides additive gains over harness-only refinement.
Analysis showing 318 evolved skills covering benchmark-specific patterns (spatial binding, attribute binding, anime style, material detail) with ~50% skill-read share on dense/compositional benchmarks.

Results¶

COMFYCLAW outperforms the no-refinement BASE across all six agent–backbone configurations on all four benchmarks (GenEval2, DPG-Bench, OneIG-EN, OneIG-ZH).
Best configuration (Claude Sonnet 4.5 + z-image-turbo): Avg 77.78 vs. BASE 67.94 (+9.84 pts); GenEval2 62.01 vs. 41.88 (+20.13 pts); DPG-Bench 91.12 vs. 89.00; OneIG-EN 80.93 vs. 71.56; OneIG-ZH 77.06 vs. 69.30.
Skill evolution contributes ~4 absolute points over verifier-only (harness without evolution), and the full system gains ~10 points over no-refinement control.
Evolved skills account for 70.0% of skill reads on DPG-Bench and 56.2% on GenEval2, but only 7.5–16.3% on the OneIG splits.
Human annotators prefer COMFYCLAW over no-skill-evolution variants on a 2,400-image study.
Claude Sonnet 4.5 produced 318 unique skills (4,768 total versioned entries) across all settings.

Limitations¶

Evaluation relies entirely on VLM-as-judge (Qwen3-VL-8B-Instruct), introducing potential circularity since the verifier and judge share the same modality.
Skill evolution effectiveness is highly benchmark-dependent: evolved skills cover 7.5–16.3% of reads on OneIG splits, suggesting limited generalization to stylized/cross-lingual prompts.
The framework is tightly coupled to ComfyUI's node-graph interface; applicability to other workflow engines is not evaluated.
Mutation validation uses only three synthesized held-out prompts per cluster, which may be insufficient to detect overfitting to cluster-specific patterns.
Infrastructure cost is high: a fresh ComfyUI server endpoint is instantiated per experimental condition.

Relevance to Vision-Language Models¶

COMFYCLAW places a VLM at the center of a closed-loop control system: it decomposes natural-language prompts into verifiable visual requirements and grounds pass/fail judgments in localized image regions, directly operationalizing VLM spatial and compositional reasoning as executable feedback signals. The work contributes to the growing VLM-as-judge paradigm by showing that region-level, requirement-decomposed VLM feedback—rather than scalar scores—enables structured workflow repair and skill distillation. For researchers tracking VLMs, the framework raises questions about verifier reliability, since the quality of the skill library is upper-bounded by the verifier VLM's own compositional and counting capabilities. It also instantiates a concrete architecture for deploying multimodal foundation models (Claude Sonnet 4.5, Qwen, Gemma) as both planners and verifiers in visual production pipelines.