Training the Orchestrator: A Supervised Approach to End-to-End PDDL Planning with LLM Agents¶

🕒 Published (v1): 2026-06-19 20:53 UTC · Source: Arxiv · link

Why this paper was selected

Supervised orchestrator training for PDDL planning; end-to-end harness-to-planner bridge

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HALO replaces the frontier-LLM orchestrator at the centre of an agentic PDDL planning harness with a small QLoRA-tuned policy trained on verifier-accepted refinement trajectories. It matches or exceeds a GPT-5-mini prompted baseline and sits within 3 pp of Gemini-3-Flash, while cutting orchestration cost ~45× and reducing total LLM calls per episode by 40–50%.

Problem¶

Agentic PDDL repair frameworks (La Malfa et al. 2026) coordinate a pool of specialised repair agents via a frontier-LLM orchestrator that is called once per refinement step. This makes orchestration the dominant cost, and locks deployment to frontier-API access—pricing out edge, lab-scale, and air-gapped settings. There is no obvious per-step ground-truth label for "correct next agent," since the only externally verifiable signal is whether the final plan is valid.

Method¶

HALO frames orchestrator training as single-token imitation learning. The key insight is that the framework's existing external validator (Fast Downward + VAL/uVAL) already provides binary correctness signals: any refinement trajectory that terminates in $V(D_T, P_T, \rho_T) = \text{valid}$ is a sequence of demonstrably correct $(s_t, a_t)$ pairs and can be used directly as supervision.

Data collection. A strong teacher (GPT-5-mini or Gemini-3-Flash) generates rollouts over 11 PDDL domains. Trajectories are filtered in three passes: (1) hard verifier filter—only trajectories ending in a valid plan survive; (2) spec-level augmentation (paraphrase NL spec, rename objects, vary goal surface) to expand ~200 seeds to ~5k trajectories; (3) soft LLM-as-judge filter on rationale coherence, yielding ~2k high-quality trajectories and ~12–15k $(s_t, a_t)$ pairs.

Policy architecture. A base LLM (Llama-3, Qwen-2.5, or Gemma-2 family) is fine-tuned with QLoRA (4-bit weights, LoRA rank $r=16$, $\alpha=32$, dropout 0.05 on attention and MLP projections). The output is a single token $a^\star \in \{0,\ldots,20\}$; loss is:

\[\mathcal{L}(\theta; s, a^\star) = -\log \pi_\theta\!\left(a^\star \mid \text{encode}(s)\right)\]

with all prompt tokens masked ($\text{label}=-100$). Greedy decoding (temperature 0) is used at inference.

Hybrid policy. Three trivially-decidable decisions (cold start → AgentEmergency; syntax error → AgentSyntaxPDDL; plan valid → NoOpAgent) are resolved by hardcoded rules (Layer 1) before the trained policy is consulted (Layer 2). Steps covered by Layer 1 are excluded from training so the model does not waste capacity on them.

Expanded agent pool. The original 13-agent pool is extended to 21 by adding five deterministic plan-repair routines (variable-swap beam search, action reorder, redundancy strip, plan truncate, subplan fill) and three PDDL-aware LLM agents (type hierarchy fixer, predicate generaliser, initial-state suggester).

Key Contributions¶

Trained orchestrator (HALO): QLoRA-tuned small model replaces a per-step frontier-LLM call inside an agentic PDDL harness, running locally on a single GPU in milliseconds per decision.
Verifier-guided trajectory supervision: Treats the framework's own compiler+validator as a binary data-acceptance gate at training time; verifier never enters the loss function.
Hybrid hardcoded+learned policy ($\pi_\text{hybrid}$): Separates trivially-decidable cases (3 rules) from genuinely ambiguous ones (trained policy), propagating this split into data collection.
Expanded 21-agent action space: Adds 5 deterministic and 3 PDDL-aware LLM agents to the baseline 13, extending coverage of modelling and plan-repair failure modes.

Results¶

Success rate vs. GPT-5-mini: HALO matches or exceeds GPT-5-mini across all 11 PDDL domains (PlanBench, Google Natural Plan, Hanoi, Blocksworld, childsnack, floortile).
Success rate vs. Gemini-3-Flash: Within 3 percentage points across benchmarks.
Orchestration cost: $0.004 vs. $0.18 per task against GPT-5-mini (~45× cheaper); ~15× cheaper than Gemini-3-Flash.
LLM calls per episode: Reduced by 40–50% relative to frontier-LLM baselines.
Training set size: ~12–15k $(s_t, a_t)$ pairs from ~200 verifier-accepted seed trajectories after augmentation and filtering.

Limitations¶

Context window is capped at 2048 tokens with per-section left-truncation; long PDDL domains or histories may lose relevant information.
Training data is generated by a single strong teacher (GPT-5-mini); diversity is partially addressed by Gemini-3-Flash but teacher-specific biases may persist.
Evaluated on 11 PDDL domains; generalization to out-of-distribution domains not demonstrated.
The soft LLM-as-judge filter is itself a frontier-LLM call, adding cost to the offline training pipeline (though not inference).
Agent IDs must tokenize to exactly one token under the active tokenizer; model families outside Llama-3/Qwen-2.5/Gemma-2 require verification passes and may not satisfy this constraint.
Tmax = 10 iteration budget; performance on problems requiring longer refinement chains is not reported.

Relevance to Harnesses / Meta-Harnesses¶

HALO is a direct contribution to the meta-harness problem: it addresses the question of what should sit at the centre of a multi-agent orchestration harness and how that controller should be trained. Rather than treating the orchestrator as a fixed, prompted frontier LLM (the current default in agentic harnesses), HALO shows that verifier-certified trajectories produced during normal harness execution are sufficient supervision signal to distill the orchestrator into a small, local model—a general recipe applicable wherever a harness contains a formal correctness oracle. The verifier-as-data-filter pattern (hard gate, not loss signal) and the hybrid hardcoded-rules + learned-policy decomposition are reusable design primitives for any harness with mixed trivially-decidable and ambiguous routing decisions.