Skip to content

PerfGuard: A Performance-Aware Agent for Visual Content Generation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PerfGuard is an LLM-agent framework for visual content generation (AIGC) that replaces vague textual tool descriptions with a structured, multi-dimensional performance boundary model, enabling data-driven tool selection at each planning step. It adds online score correction and preference-aligned planner training to keep selection accurate as tools evolve or deviate from benchmark conditions. On three benchmarks (T2I-CompBench, OneIG-Bench, Complex-Edit), it outperforms all tested baselines across generation and editing tasks.

Problem

Existing LLM agent frameworks assume tool invocations always succeed and describe tool capabilities only with coarse natural-language strings. These descriptions cannot encode fine-grained performance differences between tools (e.g., two text-to-image models with distinct strengths in spatial vs. semantic binding), introduce systematic planning errors in multi-step AIGC pipelines, and cannot adapt when actual tool behavior deviates from benchmark scores or when new tools are added.

Method

PerfGuard builds on a four-role standardized agent system (Analyst, Planner, Worker, Self-Evaluator) and introduces three tightly coupled mechanisms:

PASM (Performance-Aware Selection Modeling): Each tool is characterized by a performance boundary matrix \(M_p \in \mathbb{R}^{d \times l}\) (dimensions \(d\) × tools \(l\)) sourced from established benchmarks (T2I-CompBench for 7 generation dimensions; ImgEdit-Bench for 7 editing dimensions). For each subtask \(u_t\), the Worker generates a preference weight vector \(W_{task} \in \mathbb{R}^{1 \times d}\) and computes suitability scores \(S_{tools} = W_{task} \cdot \text{Normalize}(M_p)^\top\), then ranks tools by descending score.

APU (Adaptive Preference Updating): Combats benchmark-to-task distribution shift via an exploration-exploitation strategy: the top-\(m\) tools plus \(n\) randomly sampled tools are executed; a multimodal LLM judge produces an actual ranking \(R_{actual}\). The performance matrix is updated as \(M_p^{new} = \text{Normalize}(M_p + W_{task} \cdot \eta \cdot \Delta)\) where \(\Delta = (R_{theory} - R_{actual}) / (m+n)\). New tools without history are initialized to the category average.

CAPO (Capability-Aligned Planning Optimization): Extends Step-aware Preference Optimization (SPO) to the Planner's autoregressive subtask generation. At each step \(t\), \(k\) candidate subtasks are sampled; winning (\(u_t^w\)) and losing (\(u_t^l\)) subtasks are identified by Self-Evaluator scores. The Planner is fine-tuned via a DPO-style objective conditioned on task summary \(\tau^*\), target semantics \(s^*\), tool profiles \(B\), and execution history \(h_{t-1}\). A memory retrieval module using CLIP similarity reuses optimal subtask sequences from prior tasks as \(\beta k\) of the \(k\) candidates.

Key Contributions

  • PASM: Replaces coarse tool descriptions with standardized multi-dimensional benchmark scores as performance boundary matrices, enabling weighted, quantitative tool ranking per subtask.
  • APU: Online update rule that corrects benchmark-derived scores using observed execution rankings, with exploration to surface underrated tools and cold-start initialization for new tools.
  • CAPO: SPO-derived planner training objective that aligns subtask generation with performance-aware execution feedback through stepwise preference optimization over planning trajectories.
  • Memory retrieval in CAPO: CLIP-based retrieval of high-performing historical subtask sequences accelerates candidate generation and improves sample efficiency.
  • State-of-the-art results on T2I-CompBench, OneIG-Bench, and Complex-Edit Level-3.

Results

Basic image generation (T2I-CompBench): - PerfGuard Color: 0.8753, Shape: 0.7366, Texture: 0.8148 — best on all seven dimensions vs. FLUX, SD3, GoT, T2I-R1, GenArtist, T2I-Copilot. - Spatial: 0.6120 vs. next-best GenArtist 0.5437; Complex: 0.5007 vs. next-best GenArtist 0.4499.

Advanced image generation (OneIG-Bench): - PerfGuard Alignment: 0.834, Text: 0.684, Reasoning: 0.350 — best on all metrics. - Reasoning margin: 0.350 vs. T2I-Copilot 0.318 and T2I-R1 0.297.

Complex image editing (Complex-Edit Level-3): - PerfGuard IF: 8.95, PQ: 9.02, IP: 8.56, Overall: 8.84 — best on IF, PQ, and Overall vs. AnySD, Step1X Edit, GenArtist, OmniGen.

Ablation (T2I-CompBench subset): - CAPO alone: Color 0.8466, Spatial 0.5756, Complex 0.4493. - PASM alone: Color 0.8521, Spatial 0.5919, Complex 0.4412. - PASM+APU: Color 0.8596, Spatial 0.6005, Complex 0.4738. - Full (CAPO+PASM+APU): Color 0.8753, Spatial 0.6120, Complex 0.5007 — all components contribute additively.

Limitations

  • Tool performance boundary matrices are bootstrapped from specific benchmarks (T2I-CompBench, ImgEdit-Bench); performance on out-of-distribution tasks or novel task types not covered by these benchmarks may be miscalibrated until APU corrects them.
  • APU relies on a multimodal LLM judge for \(R_{actual}\); judge quality and cost are not analyzed—any systematic judge errors propagate into boundary score updates.
  • CAPO requires collecting planning trajectory data with winning/losing subtask pairs; computational cost of generating \(k\) candidates per step and running the Self-Evaluator is not quantified.
  • Toolset limitations are acknowledged: on OneIG-Bench alignment/text metrics, PerfGuard's margin over T2I-Copilot is small due to the ceiling imposed by available tools rather than the framework itself.
  • Only evaluated on visual content generation (AIGC); generalizability to other agent domains is unvalidated.

Relevance to Harnesses / Meta-Harnesses

PerfGuard is directly relevant as a meta-harness pattern: it wraps a heterogeneous tool library with a structured capability model (performance boundary matrices) and a runtime adaptation loop (APU), decoupling the orchestration layer from any specific tool. This is the canonical meta-harness concern—how a governing framework selects, evaluates, and updates its constituent tools without hardcoding assumptions about their reliability. The CAPO mechanism further shows how a meta-harness can feed execution quality signals back into the planner's policy, a form of closed-loop self-improvement that parallels reinforcement-style training in harness design. For researchers building harnesses over large, evolving tool ecosystems, APU's exploration-exploitation update rule offers a concrete, lightweight mechanism for keeping tool routing accurate as tools are added, updated, or drift in behavior.