Skip to content

Enhancing LLM Planning for Robotics Manipulation through Hierarchical Procedural Knowledge Graphs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HP-KG is a large-scale Hierarchical Procedural Knowledge Graph (42k+ activity steps, 6k+ household tasks) built automatically via LLM multi-agents to inject structured procedural commonsense into robot planners. Augmenting a 7B LLM with HP-KG via graph retrieval-augmented planning outperforms a 72B LLM baseline on household activity planning, while cutting inference time by ~50%.

Problem

LLM-driven robotic planners fail on complex, long-horizon manipulation tasks because they lack accurate procedural commonsense (e.g., missing prerequisite actions, ignoring physical constraints like opening a container before reaching inside). Existing remedies use flat reference documents without hierarchical structure, and large-scale LLMs (e.g., PaLM-E at 562B parameters) are energy-prohibitive for embodied deployment.

Method

HP-KG organizes procedural knowledge into three graph layers — tasks (complex composite goals), steps (simple composite procedures), and actions (atomic robot-executable primitives) — connected by typed relations (HasStep, HasAction, NextStep, NextAction, HasDescription, HasTips).

Construction pipeline (4 stages): 1. Data cleaning: filter WikiHow corpus to household categories using DeepSeek-V3; standardize BEHAVIOR-1K BDDL files into WikiHow format. 2. Procedure generation: an LLM Procedure Generator extracts tasks/steps from documents and completes atomic actions with textual attributes. 3. Iterative verification & refinement: a Rules-Guided Verifier scores each procedure against a rule set R via binary per-rule compliance checks; a Procedure Refiner modifies procedures until the score exceeds a threshold (max 3 iterations). Cosine-similarity-based redundancy clustering is applied within the verifier. 4. Hierarchical clustering & summarization: a Summary Agent clusters semantically similar nodes at each level and merges them into unified canonical procedures, applied bottom-up from task → action level.

At inference, Graph Retrieval-Augmented Planning encodes the query, retrieves top-k₁ nodes by cosine similarity, expands via K-hop BFS, filters to target level, re-ranks to top-k₂, then serializes the resulting subgraphs into textual context fed to the planner LLM.

Key Contributions

  • Novel three-layer (task / step / action) hierarchical knowledge graph structure with typed relations and safety tip attributes.
  • Automated multi-agent HP-KG construction pipeline eliminating manual knowledge engineering.
  • HP-KG dataset: 42,000+ activity steps across 6,000+ household tasks sourced from WikiHow + BEHAVIOR-1K.
  • Graph retrieval-augmented planning method (K-hop BFS + re-ranking + subgraph verbalization) enabling small LLMs to surpass larger baselines.

Results

  • ActPlan-1K (LLM planning): Qwen2-VL-7B + HP-KG (Top-3) achieves +17.64% Average LCS over 7B baseline, surpassing Qwen2-VL-72B without HP-KG (8.71 vs 8.78); 72B + HP-KG gains +11.74%.
  • ActPlan-1K: GPT-4o + HP-KG (Top-5) +3.99%; Gemini-2.0-Flash + HP-KG (Top-5) +4.90%.
  • RLBench (zero-shot manipulation): HP-KG improves VoxPoser average success rate by up to +10% (Top-5) and MA by up to +11% (Top-3) across 6 complex tasks.
  • Blocks Arrange: GPT-4o + HP-KG (Top-3) achieves 45% SR vs 33% for GPT-4o + CoT and 26% for SayCan.
  • Efficiency: Qwen2-VL-7B + HP-KG matches Qwen2-VL-72B baseline performance at ~50% inference time (0.44s vs 0.51s per sample on ActPlan-1K).
  • Ablation (graph structure): Hierarchical Graph > Coarse Graph > Unstructured Docs > no augmentation on ActPlan-1K.
  • Ablation (pipeline stages): each stage (generator → verifier → summary agent) contributes incremental LCS gains.

Limitations

  • HP-KG is restricted to household activities; generalizing to other domains (industrial, outdoor, surgical) requires new data collection and reprocessing.
  • Retrieval quality degrades for smaller models at higher Top-K due to limited long-context comprehension.
  • Construction pipeline relies on GPT-4o for verification/summarization, incurring API cost and availability dependency.
  • Evaluation on a narrow set of 6 RLBench tasks; broader task coverage not reported.

Relevance to Agentic AI / LLM Agents

HP-KG directly addresses a core bottleneck in LLM-based agentic systems: grounding high-level language instructions in physically executable, sequenced actions. The automated multi-agent construction pipeline (Generator → Verifier → Refiner) is itself an instance of multi-agent collaboration for knowledge curation, relevant to anyone building self-improving or tool-augmented agent pipelines. The graph retrieval-augmented planning paradigm extends RAG from factual recall to procedural reasoning, offering a structured alternative to pure in-context prompting for long-horizon agentic tasks. The result that a 7B agent augmented with external procedural knowledge outperforms a 72B agent without it directly challenges the assumption that scaling model size is the primary lever for capable embodied agents.