Multi-View Encoders for Performance Prediction in LLM-Based Agentic Workflows¶

🕒 Published (v1): 2025-05-26 09:46 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Agentic Predictor is a lightweight performance predictor for LLM-based agentic workflows that encodes workflows via three complementary views—graph structure, code, and system prompts—and uses cross-domain unsupervised pretraining to reduce the labeled data required for training. It replaces expensive trial-and-error LLM evaluations during automated workflow search with fast surrogate predictions. Across three benchmark domains, it improves prediction accuracy by up to 6.90% and workflow utility by up to 5.87% over strong graph-based baselines.

Problem¶

Automated agentic workflow design requires evaluating large numbers of candidate configurations, each demanding repeated LLM API calls that are costly and slow. Existing approaches either hand-craft workflows or use exhaustive search (e.g., REINFORCE on DAGs, LLM-driven iterative synthesis), all of which incur high computational and financial overhead. No prior work addresses the dual challenge of workflow heterogeneity (subtle configuration variations causing dramatic performance differences) and label scarcity (prohibitive cost of generating ground-truth performance labels).

Method¶

Agentic Predictor models a workflow as \(W = \{V, E, P, C\}\) (agents, edges, prompts, code), represented as a DAG, and learns to predict task success rate \(\hat{e} = M_\Theta(W, T)\) without executing the workflow.

Multi-view encoding produces three modality-specific representations fused into \(Z \in \mathbb{R}^d\):

Graph encoder: Three GNN-processed graphs (\(G_{\text{prompt}}, G_{\text{code}}, G_{\text{operator}}\)) share topology but differ in node features; cross-view self-attention (\(\hat{X} = \text{LN}(\text{MHA}(X,X,X)+X)\)) followed by view-attention pooling and graph readout yields \(Z_G\).
Code encoder: An \(L\)-layer MLP over the full workflow code string yields \(Z_C\).
Prompt encoder: A separate \(L\)-layer MLP over concatenated system/instruction prompts yields \(Z_P\).
Aggregation: \(Z = \text{MLP}([Z_G, Z_C, Z_P])\).

Cross-domain unsupervised pretraining (Agentic Predictor+) trains the encoder on unlabeled workflows via a joint reconstruction + contrastive loss:

\[\mathcal{L}_{\text{enc}} = \mathcal{L}_{\text{rec}} + \mathcal{L}_{\text{con}}\]

where \(\mathcal{L}_{\text{rec}}\) is an \(\ell_2\) reconstruction of each modality embedding, and \(\mathcal{L}_{\text{con}}\) is a cross-modal InfoNCE contrastive loss over view pairs \((G,C)\), \((G,P)\), \((C,P)\).

Performance predictor: A task encoder (T5/BERT) produces task embedding \(T\); the joint representation \(F = [Z, T]\) feeds a lightweight MLP head trained on a small labeled set with binary cross-entropy or MSE loss. At inference, the predictor ranks \(K\) sampled candidates, replacing full LLM execution.

Key Contributions¶

Multi-view workflow encoding combining graph, code, and prompt views with cross-view attention and view-attention pooling.
Cross-domain unsupervised pretraining (reconstruction + contrastive) to mitigate labeled data scarcity.
Search-agnostic predictor-guided workflow search that replaces expensive LLM-call evaluations with fast surrogate scoring.
A curated benchmark spanning three domains for evaluating agentic workflow performance predictors.
Empirical demonstration of up to 6.90% accuracy and 5.87% utility improvement over graph-only baselines.

Results¶

Averaged across three benchmark domains, Agentic Predictor improves prediction accuracy by up to 6.90% and workflow utility by up to 5.87% over strong graph-based baselines.
Outperforms single-view GNN-based predictor (FLORA-Bench baseline) on both accuracy and utility metrics.
Agentic Predictor+ (with unsupervised pretraining) achieves better low-label-regime generalization than the supervised-only variant.
Baselines compared include GNN-based predictors and MAS-GPT-style LLM-finetuned generators; Agentic Predictor is the only system with multi-view encoding, unsupervised pretraining, and search-agnostic lightweight inference simultaneously (Table 1).

Limitations¶

Text is truncated before full experimental results are reported; specific per-domain numbers and ablation tables are not available in the provided excerpt.
Predictor quality depends on the quality and diversity of the unlabeled pretraining corpus; domain shift between pretraining and target domains is not deeply analyzed in the provided text.
The predictor treats workflow performance as a fixed scalar or binary label, potentially missing the sensitivity of performance to specific query distributions or few-shot examples.
The predictor-guided search is instantiated as random search + ranking; the interaction between predictor quality and more sophisticated search algorithms (e.g., evolutionary, Bayesian) is deferred to an appendix (§B.6).
Encodes the full workflow code as a flat string (MLP), which may fail to capture fine-grained structural differences in large or deeply nested codebases.

Relevance to Harnesses / Meta-Harnesses¶

Agentic Predictor directly addresses a core bottleneck in meta-harness design: the cost of evaluating candidate workflow configurations during automated orchestration. By learning a surrogate model over the configuration space of agentic pipelines—encoding topology, code logic, and prompts jointly—it provides a principled mechanism for a meta-harness to prune the search space before committing to expensive LLM execution. The cross-domain pretraining strategy is especially relevant for meta-harnesses that must generalize across heterogeneous task domains without retraining from scratch. The search-agnostic design means this predictor can slot into any existing meta-harness optimization loop as a drop-in evaluator, reducing per-iteration cost from \(O(\text{LLM calls})\) to \(O(\text{forward pass})\).