Skip to content

InstructFlow: Adaptive Symbolic Constraint-Guided Code Generation for Long-Horizon Planning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InstructFlow is a multi-agent framework for robotic manipulation that uses a hierarchical instruction graph and symbolic constraint induction to generate and repair executable code for long-horizon tasks. When execution fails, a Constraint Generator abstracts the failure into logical predicates, which propagate upstream to refine the graph and trigger targeted code revision rather than full regeneration. Across drawing, block-stacking, and YCB packing benchmarks, it outperforms prior LLM-based planners by 20–40% in task success rate.

Problem

LLM-based robotic code generators struggle with three compounding failures: (1) decomposing long-horizon tasks into physically feasible subtasks, (2) enforcing spatial/temporal/physical constraints reliably, and (3) recovering adaptively from execution failures—they typically resort to blind retries or wholesale replanning that reproduces the same errors. Existing feedback-driven methods (e.g., PRoC3S) react to failures at the surface level without diagnosing structural causes.

Method

InstructFlow coordinates three agents over a shared hierarchical instruction graph G = (V, E):

  1. InstructFlow Planner — builds the graph by partitioning nodes into planning nodes (grounded subgoals → robot actions) and reasoning nodes (typed symbolic transformations: spatial relation, density, object selection, action ordering, parameter refinement). The graph is reconstructed each feedback round, inserting reasoning nodes upstream of failing subgoals.

  2. Code Generator — synthesizes Python code per planning node by encoding its subgoal enriched with symbolic outputs from upstream reasoning nodes into structured prompts.

  3. Constraint Generator — runs a four-stage failure diagnosis: (i) retrieve failure-relevant entities from execution traces; (ii) instantiate symbolic predicates over physical variables (poses, distances); (iii) compute geometric diagnostics (collision proximity, path clearance, placement stability); (iv) induce a symbolic constraint ϕ as a conjunction of relational constraints R_i(e_a, e_b) and physical feasibility conditions f_j(Θ_j) ⊕ τ_j.

The induced constraints Φ are fed back to the Planner, which inserts only the reasoning nodes needed to repair the failing subgoals, leaving unaffected nodes unchanged—avoiding full plan regeneration.

Key Contributions

  • Modular multi-agent architecture with a dynamic, graph-guided prompt composition loop that treats symbolic feedback as a first-class signal rather than a re-prompt trigger.
  • Symbolic constraint induction mechanism: abstracts raw execution failures into reusable logical predicates (e.g., ClearOf, ProximitySafe, StableOn) grounded in four component types—entities (E), relations (R), physical functions (F), thresholds (B).
  • Targeted subgoal repair: constraints propagate to only the affected instruction graph nodes, enabling localized code revision without full regeneration and improving sample efficiency.
  • Empirical validation on three Ravens simulation domains (Drawing, Arrange-Blocks, Arrange-YCB) with robustness testing under perceptual noise and feedback ambiguity.

Results

  • Overall: InstructFlow outperforms all baselines by 20–40% in task success rate across benchmarks (GPT-4o backbone, 10 seeds per task, max 1000 samples/trial).
  • Drawing: 100% on Star, Arrow, Enclosed; 80% on Letters. PRoC3S achieves 90/80/80/90%; LLM³ 40/40/80/50%.
  • Arrange-Blocks: Pyramid 90% (vs. PRoC3S 60%), Line 100% (vs. 70%), Packing 90% (vs. 50%), Unstack 90% (vs. 60%).
  • Arrange-YCB: Packing 60% (vs. PRoC3S 30%), Stacking 70% (vs. 40%).
  • Ablation: removing the Planner agent drops success by up to 50% on spatial tasks; removing the Constraint Generator drops 30–40% on cluttered/precision tasks.
  • Perceptual noise (σ=0.02, ~50% object size): average drop of 18.3%; most tasks remain above 70%.
  • Feedback noise (incorrect/incomplete traces): average drops of 16% and 11% respectively.

Limitations

  • Evaluated exclusively in Ravens simulation (UR5 + PyBullet); no real-robot experiments.
  • Constraint induction relies on structured failure traces from a deterministic CCSP checker—noisy or unstructured real-world sensors degrade performance (16% drop under incorrect feedback).
  • Feedback loop capped at 5 iterations; tasks requiring deeper repair chains may not converge.
  • No visual/multi-modal grounding; symbolic predicates are derived from pose data, not raw perception.
  • All experiments use GPT-4o; generalization to smaller or open-weight LLMs is untested.

Relevance to Agentic AI / LLM Agents

InstructFlow directly addresses a core bottleneck for tool-using LLM agents: structured, interpretable failure recovery without fine-tuning. The symbolic constraint induction mechanism—translating execution traces into reusable logical predicates that refine a hierarchical task graph—is an architecture pattern applicable well beyond robotics, relevant to any agentic pipeline where actions have verifiable preconditions and postconditions. The graph-guided prompt composition, where symbolic feedback is progressively injected into downstream agent prompts, is a concrete implementation of "deliberate, compositional reasoning" as opposed to stateless re-prompting. This work connects to the broader line of neuro-symbolic agent architectures that seek to give LLM planners persistent, structured world models for multi-step decision-making.