Skip to content

Design and Implementation of Agentic Orchestrations and Orchestration of Agents

🕒 Published (v1): 2026-06-30 11:31 UTC · Source: Arxiv · link

Why this paper was selected

Rinderle-Ma (U Vienna, BPM authority); directly about designing agent orchestration — the core harness problem

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a four-option classification framework (OO1–OO4) for agentic orchestration that spans from fully autonomous, process-agnostic LLM agents to explicitly framed multi-agent process orchestrations. It implements all four options for a cyber-physical light sensing scenario using MCP-connected LLMs and proposes quantitative metrics (adapted Cyclomatic complexity, ABC complexity, precision/recall, FNR, reactivity speed) to assess each option. Results show that process-framed orchestrations (OO4) achieve near-perfect correctness (\(F_1 = 0.97\)) while unframed agents (OO1) achieve only \(F_1 = 0.40\).

Problem

Agentic BPM lacks principled criteria—both qualitative and quantitative—for choosing how much control logic to encode in a surrounding process frame versus leaving it to LLM agent autonomy. Existing manifestos define agents and research agendas but do not address realization tradeoffs or provide metrics for comparing implementations at different autonomy-vs.-traceability operating points.

Method

The authors define four Orchestration Options (OO): - OO1 – Process-agnostic agentic orchestration: one LLM agent drives all steps via MCP tool calls, no explicit process frame. - OO2 – Process-aware agentic orchestration: one agent operates under an internalized frame (rules injected into the prompt), still no external process engine. - OO3 – Orchestration of process-agnostic agents: a deterministic process engine (CPEE) invokes agents for sub-tasks; agents receive minimal context. - OO4 – Orchestration of process-aware agents: CPEE frame orchestrates agents that are themselves given rule-aware prompts.

Each OO is implemented for a predictive light sensing scenario (lux/occupancy/movement sensor via REST/MQTT exposed as MCP tools) using four LLMs (Ministral-3B, Gemma-4-31B, Qwen3.6-35B, qwen-35b-coding) against 9-rule ground truth (Table 2).

Quantitative metrics proposed: - Task specificity: Cyclomatic complexity \(M = E - N + 2P\) and ABC complexity \(|ABC| = \sqrt{A^2 + B^2 + C^2}\) measured over the surrounding process model. - Correctness: standard \(P\), \(R\), \(F_1\) against ground-truth rule firings. - Reactivity: False Negative Rate \(FNR = \frac{FN}{FN+TP}\) and mean response latency \(R_s\). - Traceability: \(F_1\) of agent-produced log \(L_o\) against actual execution log \(L_t\).

Key Contributions

  • A four-way classification of agentic orchestration options (OO1–OO4) along five properties: autonomy, task specificity, reactivity, correctness assurance, traceability/tractability.
  • Qualitative selection-criteria table mapping scenario properties (goal complexity, human oversight, constraint adherence, decision space) to the appropriate OO.
  • First set of quantitative metrics for comparing orchestration options: \(M\), \(|ABC|\), \(|ABC_w|\) for task specificity; \(P/R/F_1\) for correctness; \(FNR\)/\(R_s\) for reactivity; log-\(F_1\) for traceability.
  • Reference open-source implementations of OO1–OO6 (including deterministic baselines OO5/OO6) on CPEE with MCP tool integration.
  • Practical engineering observations (7 lessons) on temporal enforcement, context window limits, logging reliability, and debugging complexity.

Results

From Table 3 (light sensing scenario, 72-second simulation):

| OO | \(M\) | \(|ABC|\) | \(P\) | \(R\) | \(F_1\) | \(FNR\) | \(R_s\) (s) | |----|-----|---------|-----|-----|--------|--------|-----------| | OO1 (process-agnostic agent) | 2 | 1.00 | 0.67 | 0.33 | 0.40 | 0.67 | 12.10 | | OO2 (process-aware agent) | 2 | 1.00 | 0.63 | 1.00 | 0.77 | 0.00 | 0.10 | | OO3 (frame + agnostic agents) | 13 | 4.47 | 0.50 | 1.00 | 0.67 | 0.00 | 10.18 | | OO4 (frame + aware agents) | 13 | 5.48 | 0.95 | 1.00 | 0.97 | 0.00 | 22.10 | | OO5 (deterministic baseline) | 17 | 10.25 | 1.00 | 1.00 | 1.00 | 0.00 | 0.08 | | OO6 (deterministic + agent UX) | 20 | 10.67 | 1.00 | 1.00 | 1.00 | 0.00 | 0.08 |

  • OO4 achieves near-ground-truth correctness (\(F_1 = 0.97\)) while retaining user-input reactivity absent from OO5.
  • OO1 has worst recall (0.33) and highest FNR (0.67), confirming that unframed agents miss required actions.
  • OO2's fast \(R_s = 0.10\) s reflects in-prompt rule adherence without process-engine overhead.
  • \(M\) and \(|ABC|\) correctly rank task specificity: OO1/OO2 score lowest, OO5/OO6 highest.

Limitations

  • Autonomy and tractability metrics are explicitly deferred to future work.
  • \(M\) and \(|ABC|\) are purely comparative within the same workflow; no absolute or cross-process semantic meaning.
  • LLM reasoning latency is non-deterministic, making temporal constraints (e.g., "check every second") unenforceable without external process frames or tools.
  • Context window degradation over long-running workflows is identified but not quantitatively addressed.
  • Single illustrative scenario (light sensing); generalizability to complex enterprise processes is only sketched (blood donation scenario), not evaluated.
  • The paper does not report traceability log-\(F_1\) results despite defining the metric.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct theoretical and empirical treatment of harness design: OO3 and OO4 map precisely to the "outer deterministic orchestrator wrapping inner LLM agents" pattern central to meta-harness architectures, and the paper gives designers a vocabulary (OO1–OO4) and metrics (\(M\), \(|ABC|\), \(F_1\), \(FNR\)) for deciding how much control logic to push into the harness versus agent prompts. The key finding—that temporal constraints and correctness guarantees require deterministic frames (OO3/OO4) rather than LLM self-regulation—directly informs how harness engineers should partition responsibility. The seven practical engineering observations (logging reliability, context window limits, debugging complexity scaling with unframedness) constitute actionable design guidelines for anyone building production harnesses.