InstructFlow: Adaptive Symbolic Constraint-Guided Code Generation for Long-Horizon Planning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InstructFlow is a three-agent orchestration framework for robotic manipulation that decomposes long-horizon tasks into a hierarchical instruction graph and, upon execution failure, induces symbolic constraints from failure traces to surgically repair only affected subgoals. It outperforms strong LLM-based planners by 20–40% in task success rate across drawing, block-stacking, and YCB-packing benchmarks.

Problem

LLM-based robotic planners fail in three compounding ways on long-horizon tasks: (1) flat plan representations lack the hierarchical structure needed for reliable subtask decomposition; (2) physical and spatial constraints are not explicitly enforced during code synthesis; (3) failure recovery defaults to blind retries or full regeneration, because models can detect failure but cannot identify its structural cause and perform targeted repair.

Method

InstructFlow orchestrates three cooperative agents sharing a central instruction graph \(G_t = (V_t, E_t)\), \(V_t = V_{plan} \cup V_{reason}\):

InstructFlow Planner constructs \(G_t\) conditioned on task goal, environment state, and prior symbolic constraints. Planning nodes \(v^{plan}\) encode typed executable subgoals (pick, place, …). Reasoning nodes \(v^{reason}\) perform five typed symbolic transformations: \(T_{spatial}\), \(T_{density}\), \(T_{select}\), \(T_{plan}\), \(T_{param}\); their outputs are injected downstream into planning-node prompts. On failure, the planner inserts reasoning nodes upstream of the affected subgoal rather than rebuilding the full graph.

Code Generator translates each planning node into Python via a structured prompt: $\(\text{code}^{(t)} = \text{LLM}\!\left(\text{Encode}\!\left(\{v^{reason}_{T_j}\}_{j=1}^{|v^{reason}(t)|}, v^{plan}_{(t)}\right)\right)\)$

Constraint Generator runs a four-stage failure-diagnosis workflow on each execution failure: (i) retrieve failure-relevant entities from trace \(F_t\) and code \(P_t\); (ii) instantiate symbolic predicates over those entities; (iii) compute geometric/physical diagnostics (e.g., \(\text{Below}(o_4, o_5) := z_4 < z_5\)); (iv) induce a symbolic constraint $\(\phi := \bigwedge_{c \in \mathcal{C}(\mathcal{E},\mathcal{R},\mathcal{F},\mathcal{B})} c, \quad \mathcal{C} = \{R_i(e^a_i, e^b_i)\} \cup \{f_j(\Theta_j) \oplus \tau_j\}\)$ as a conjunction of relational constraints and physical feasibility conditions. \(\phi\) is propagated upstream to refine \(G_t\) and re-prompt only affected nodes; unaffected subgoals are not regenerated.

Execution is evaluated by a CCSP module checking four constraint categories: kinematic reachability, collision avoidance, grasp stability, placement validity. Feedback iterations are capped at 5; sample budget per trial is 1000 (10000 for drawing). Backbone LLM: GPT-4o.

Key Contributions

• Hierarchical instruction graph \(G_t = (V_{plan} \cup V_{reason}, E_t)\) as a shared inter-agent data structure enabling structured, interpretable task decomposition with adaptive symbolic reasoning injection.
• Symbolic constraint induction mechanism that abstracts execution failures into reusable logical predicates \(\phi\) (relational + physical), enabling localized plan repair instead of full regeneration.
• Modular three-agent harness with a feedback-driven information flow operating at both graph level (reasoning → planning nodes) and prompt level (goal + constraints → structured code prompts).
• Empirical validation on Ravens/PyBullet across Drawing, Arrange-Blocks, and Arrange-YCB benchmarks showing 20–40% gains over prior methods.

Results

All numbers are task success rates over 10 randomized seeds; backbone is GPT-4o.

Drawing tasks: - Star: InstructFlow 100% vs. PRoC3S 90%, LLM3 40%, CaP 10% - Arrow: 80% vs. 80%, 40%, 0% - Letters: 100% vs. 80%, 80%, 40% - Enclosed: 100% vs. 90%, 50%, 30%

Arrange-Blocks: - Pyramid: 90% vs. 60%, 0%, 20% - Line: 100% vs. 70%, 40%, 20% - Packing: 90% vs. 50%, 30%, 20% - Unstack: 90% vs. 60%, 0%, 10%

Arrange-YCB: - Packing: 60% vs. 30%, 0%, 30% - Stacking: 70% vs. 40%, 10%, 10%

Overall improvement over baselines: 20–40 percentage points in success rate, with the largest gains on constraint-sensitive and long-horizon scenarios (Arrange-Blocks, Arrange-YCB).

Limitations

• Evaluated entirely in simulation (Ravens/PyBullet); no real-robot transfer experiments.
• Symbolic predicate vocabulary (\(\mathcal{E}, \mathcal{R}, \mathcal{F}, \mathcal{B}\)) is hand-designed for tabletop manipulation; generalization to novel failure modes or domains requires manual extension.
• Feedback iteration budget is fixed at 5; behavior when failures persist beyond this cap is not analyzed.
• Only 10 seeds per task — limited statistical power for reporting confidence in results.
• Computational cost of GPT-4o calls across three agents per iteration and per failure is not quantified.
• Constraint induction is LLM-driven; failure modes where the LLM misdiagnoses the physical root cause are not characterized.

Relevance to Harnesses / Meta-Harnesses

InstructFlow is a concrete domain instantiation of a multi-agent harness: it coordinates three specialized agents via a shared dynamic data structure (the instruction graph), with typed inter-agent communication and a feedback loop that drives selective re-execution of only the affected pipeline stages. The Constraint Generator acts as an autonomous monitor agent that produces structured correction signals — analogous to a meta-harness supervisor that introspects sub-agent outputs and routes refinement work back upstream without restarting the full pipeline. The localized repair strategy (update only the affected subgraph, not the full plan) directly maps to incremental re-execution semantics increasingly common in agentic harness designs, making InstructFlow a compelling case study for failure-aware, feedback-driven orchestration architectures.