Skip to content

GLOW: Graph-Language Co-Reasoning for Agentic Workflow Performance Prediction

🕒 Published (v1): 2025-12-11 13:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GLOW is a performance predictor for agentic workflows (AWs) that avoids costly execution-based evaluation by combining a graph-structure-aware GNN with an instruction-tuned "graph-oriented LLM" in a dual-branch architecture. It predicts whether a given AW will succeed on a task, enabling automatic AW generation frameworks to replace expensive LLM execution with a cheap surrogate. Integrated into AFLOW, GLOW reduces computation time by 98.7% with only a 0.031 average score drop.

Problem

Automatic AW generation methods (e.g., AFLOW, ADAS) evaluate candidate workflows by executing them with LLMs, which is expensive and slow, bottlenecking large-scale search. Existing surrogate predictors use GNNs on DAG-structured AWs but treat agent prompts as shallow text embeddings, failing to capture deep semantic logic. Pure LLM approaches, conversely, cannot model DAG topology or error propagation paths.

Method

GLOW encodes each AW through three parallel branches fused by a transformer encoder:

  1. Structural branch (GNN): Agent prompts are embedded with sentence-BERT as initial node features; a 2-layer GAT propagates messages along DAG edges, and mean pooling yields a global structural representation R_GNN. The GNN is pre-trained with self-supervised node-embedding reconstruction (MSE) and edge-existence prediction (BCE).

  2. Semantic branch (graph-oriented LLM): The AW is linearized into descriptive text (nodes→prompt dict, edges→tuple list). A Qwen3-1.7B model fine-tuned via QLoRA on six graph-reasoning task types (degree prediction, neighbor extraction, prompt retrieval, reachability/path length, key node identification, topological sorting) encodes this text; the final token's hidden state is projected via MLP to yield R_LLM.

  3. Task encoding: The task instruction is encoded by the same sentence-BERT and projected to R_Task.

A learnable [Pred] token is concatenated with [R_LLM; R_GNN; R_Task], augmented with learnable type embeddings, and processed by a multi-layer transformer fusion module. The [Pred] token's final hidden state is decoded by an MLP+sigmoid to a scalar score. Training combines BCE prediction loss with triplet contrastive loss applied separately to both GNN and LLM representation spaces (λ=1, α=0.2).

Key Contributions

  • Graph-oriented LLM instruction tuning: Constructs a 23,346-sample corpus of graph-reasoning QA pairs from AW DAGs; fine-tunes Qwen3-1.7B to near-perfect graph comprehension (99.1% avg accuracy vs. 65.9% zero-shot).
  • Dual-branch representation learning: Simultaneous structural (GNN) and semantic (LLM) encoding projected into a shared latent space via MLP projectors.
  • Contrastive alignment: Triplet loss clusters successful AWs and repels unsuccessful ones in both representation spaces, sharpening discriminative power.
  • Multi-stage training: GNN self-supervised pre-training → LLM instruction tuning → end-to-end fine-tuning with frozen SBERT and LLM.
  • FLORA-Bench evaluation: State-of-the-art on all six domain splits across both accuracy and ranking utility metrics.

Results

  • vs. best baseline (Agentic Predictor, AP): +1.5% accuracy, +2.0% utility on average across six FLORA-Bench domains (coding/math/reasoning × two AW generators G-Designer/AFLOW).
  • Specific domain peaks: Reason-AF utility: 90.5% (GLOW) vs. 88.1% (AP); Math-GD accuracy: 64.4% vs. 62.9%.
  • Instruction tuning: Graph-oriented LLM achieves 99.1% avg graph-task accuracy vs. 65.9% base Qwen3-1.7B; TSORT goes from 21.5% → 99.0%.
  • Ablation: Removing R_GNN causes the largest drop (−2.2% accuracy, −2.4% utility avg); removing R_LLM causes −1.2% accuracy, −2.0% utility.
  • AFLOW integration (RQ6): GLOW reduces AW generation wall-clock time by 98.7% vs. ground-truth execution; final AW score decreases by only 0.031 on average over HumanEval, MBPP, MMLU.

Limitations

  • Evaluated exclusively on FLORA-Bench; generalization to AW topologies from other generators or domains is untested.
  • The contrastive loss requires per-task triplet construction, which may be impractical when tasks have very few successful or unsuccessful AWs.
  • The graph-oriented LLM (Qwen3-1.7B) and instruction-tuning corpus are derived from the same AW pool as downstream evaluation; out-of-distribution graph structures may degrade graph comprehension.
  • Pre-training the GNN and instruction-tuning the LLM add significant setup complexity and cost before end-to-end training begins.
  • Predicts binary success/failure; does not estimate continuous performance metrics (e.g., partial scores) that some AW tasks may require.

Relevance to Harnesses / Meta-Harnesses

GLOW directly addresses a core scalability bottleneck in automated AW harnesses: the cost of evaluating candidate workflows during search. By replacing LLM-execution-based fitness evaluation with a learned surrogate, GLOW enables meta-harnesses like AFLOW (which uses MCTS over AW space) to scale candidate exploration by ~100× without rerunning the full pipeline. This is structurally analogous to neural architecture search surrogates in NAS meta-harnesses—the predictor becomes a plug-in component that gates which candidates proceed to expensive full evaluation. Researchers building or benchmarking AW generation harnesses should consider GLOW-style surrogates as a standard efficiency layer, particularly for harnesses that perform many-sample search over structured workflow spaces.