Skip to content

Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Ground-Compose-Reinforce (GCR) is a neurosymbolic RL framework that trains agents to execute temporally structured tasks specified as Reward Machines (RMs) using only 350 labelled pretraining trajectories—no manually designed reward functions or oracle labelling functions required. It grounds atomic propositions from data, composes them to interpret arbitrary RM tasks, and uses self-generated dense rewards for RL. Compositional reward shaping via learned primitive value functions (PVFs) overcomes propositional sparsity in long-horizon tasks.

Problem

Grounding language in perception and action for situated agents traditionally requires either hand-engineered domain-specific reward functions/success detectors, or massive trajectory datasets (as used by VLA models like π0, RT-2). Both approaches fail in data-scarce settings or on complex/out-of-distribution tasks. The core gap is: how to faithfully elicit complex, temporally structured behaviours from high-level language-like specifications with minimal labelled data and no external oracles.

Method

GCR has two phases operating over a fixed propositional vocabulary AP and pretraining dataset D of labelled trajectories:

  1. Ground: Train a neural labelling function L̂(s) ≈ L(s) via binary classification on D, mapping environment states to truth assignments over AP. Also learn 2|AP| primitive value functions (PVFs)*—one per proposition literal 3x and 3¬x—via offline RL on D, capturing how close any state is to satisfying each atomic proposition.

  2. Compose + Reinforce: Given a new RM task R, approximate its optimal value function (OVF) by decomposing transitions into logical subtasks, reducing to DNF clauses, and applying fuzzy-logic operators (max for disjunction, min for conjunction) over PVFs. This approximation feeds potential-based reward shaping that provides dense signals during RL (PPO), with the agent querying LÌ‚ to simulate RM state and self-generate rewards. No external reward or labelling function is accessed during training.

The result: 2|AP| pretrained PVFs bootstrap reward shaping for doubly-exponentially many (2^2^n) logical tasks and infinitely many RM tasks.

Key Contributions

  • End-to-end neurosymbolic framework (GCR) training RL agents from RM specifications without oracle reward/labelling functions
  • Compositional reward shaping via PVFs and fuzzy-logic OVF approximation, addressing propositional sparsity in long-horizon robotic tasks
  • Empirical demonstration of OOD compositional generalization: behaviours never seen in D (e.g., handling 3 boxes when D contains at most 1-box interactions) are successfully elicited
  • Natural language interface via autoformalization: GeoGrid RMs autoformalized zero-shot from NL using o3, enabling NL-driven agent control

Results

All numbers are undiscounted return averaged over 5 runs, evaluated under ground-truth L*. Baselines: LTL-BC, Bespoke Reward Model, Bespoke BC (all have advance task knowledge; GCR does not).

GeoGrid (5000 random-policy trajectories): - Sequence: GCR 1.00 ± 0.00 vs. all baselines ≤ 0.05 - Loop: GCR 5.36 ± 0.08 (matches theoretical max, exceeds best D trajectory of 3) vs. all baselines ≤ 0.04 - Logic: GCR 0.94 ± 0.01 vs. all baselines = 0 - Safety: GCR 1.00 ± 0.00 vs. best baseline −0.14

DrawerWorld (350 hand-collected trajectories): - Hold-Red-Box: GCR 1538 ± 130 (exceeds best D trajectory of 736) vs. all baselines = 0 - Pickup-Each-Box: GCR 1.00 ± 0.00 (OOD: D contains ≤1-box interactions) vs. all baselines = 0 - Show-Green-Box: GCR 0.61 ± 0.06 vs. all baselines = 0

Reward shaping ablations: removing shaping (No RS) collapses performance on DrawerWorld to 0; high-level RS (RM-state-only potential) partially recovers GeoGrid but still 0 on DrawerWorld manipulation tasks.

Limitations

  • Requires a fixed, pre-defined propositional vocabulary AP with labelled trajectory data; acquiring labels at scale (crowdsourcing, self-supervised) is deferred to future work
  • PVF composition via fuzzy logic introduces approximation error; expressivity is traded for modularity
  • Reward hacking risk remains: misalignment between LÌ‚ and L* can lead to policies that exploit the learned reward model (evidenced in Bespoke Reward Model baseline; GCR partially mitigates but doesn't eliminate)
  • Evaluated on relatively small, controlled domains (2D gridworld, Meta-World subset); scaling to pixel-based or real-robot settings undemonstrated
  • NL interface depends on an external autoformalizer (o3); zero-shot autoformalization accuracy and failure modes are not systematically evaluated
  • Framework assumes RM tasks can be compositionally expressed over AP; non-compositional or underspecified NL goals are out of scope

Relevance to Agentic AI / LLM Agents

GCR directly addresses the data efficiency bottleneck in building language-driven agents: where VLA models require internet-scale trajectory data, GCR achieves compositional generalization from 350 trajectories by exploiting formal task structure. The RM-as-task-specification paradigm offers a middle path between brittle manual reward engineering and data-hungry end-to-end learning—relevant to agent frameworks that need verifiable, interpretable task representations. The autoformalization result (NL→RM via o3, zero-shot) suggests a practical interface where LLMs handle language understanding and RL handles grounded execution, cleanly separating concerns. For researchers building agents that must reliably follow temporally structured instructions in low-data settings (robotics, tool-use agents), the compositional reward shaping mechanism and PVF pretraining are directly transferable techniques.