Skip to content

Hierarchical Experimentalist Agents

🕒 Published (v1): 2026-06-28 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HExA (Hierarchical Experimentalist Agents) is a training-free, in-context RL framework in which LLM agents actively experiment with a tool-augmented environment, distill experience into a persistent hierarchical skill bank, and reuse those skills across episodes and task families. On Interphyre—a new 2D physics benchmark the authors introduce—HExA improves Claude Sonnet 4.6 from 2% to 77% success on the hardest levels, and achieves 44% via zero-shot cross-level skill transfer alone.

Problem

LLMs trained on static corpora fail in novel environments whose dynamics were never seen in training: parametric recall, chain-of-thought, and RAG all supply only fixed prior knowledge, not instance-specific evidence gathered from interaction. Even when general principles are encoded (e.g., laws of physics), applying them precisely to a specific novel configuration requires active experimentation that static retrieval cannot substitute. Existing in-context RL methods require external feedback signals, oracle supervision, or parameter access—none of which is available with closed-source models in genuinely open-ended domains.

Method

HExA couples three components in a looped meta-episode over a task family \(\ell\) with seeds \(S_\ell = \{s_1, \dots, s_{|S_\ell|}\}\):

Actor \(\pi_\text{act}\): any tool-augmented LLM. At episode start it receives a skill context \(G = \text{RETRIEVER}(K, \ell, M, N)\)—the top-\(M\) skills and top-\(N\) mistake records from the current bank—prepended to its system prompt. It then interacts with the environment via tool calls (scene inspection, partial simulation, contact logging, counterfactual snapshot/restore) for up to \(T\) turns, producing trajectory \(\tau_j\).

Reward: each trajectory earns \(r(\tau) \in [-1, +1]\) encoding both outcome (success/failure) and efficiency (speed relative to budget \(T\)), binned into qualitative categories (fast success, slow success, exploratory failure, early exit) so the evolver LLM can act on them reliably.

Evolver \(\pi_\text{ev}\): after each round of \(x\) trajectories, receives the current bank \(K_{n-1}\) plus the new reward-tagged batch \(T^{(n)}\). It contrasts high-reward and low-reward trajectories to extract (a) strategy skills—structured records of (title, principle, when-to-apply, example, reward score \(r_k\)) abstracted from successful and partially successful trajectories—and (b) mistake records \(\mu = (\delta, \rho, \alpha)\) of failure description, root cause, and corrective action. The bank is capped at \(M_\text{max}\) skills and \(N_\text{max}\) mistakes to bound context injection size.

Cross-task transfer: given evolved source banks \(K_{\ell_1}, \dots, K_{\ell_S}\) and only a textual description of unseen target \(\ell^*\), the evolver synthesizes \(K_{\ell^*}\) by selecting structurally relevant source skills, stripping source-specific parameter bindings, re-grounding each mechanism in the target's entities and constraints, and assigning a transfer score by structural correspondence and cross-source corroboration. No target-task trajectories are required.

Key Contributions

  • Formalizes experiment-centric in-context learning: a problem setting where solving an instance requires active environmental interaction rather than recall and reasoning alone.
  • Introduces Interphyre, a 2D physics benchmark extending PHYRE with programmatic tool-call and intervention APIs (scene inspection, partial simulation, contact logging, counterfactual snapshot/restore) for evaluating and training experimentalist agents.
  • Proposes HExA, a training-free actor–evolver–retriever loop that builds a hierarchical, composable skill bank from interaction experience, compatible with any black-box LLM, requiring no parameter updates or external supervision.
  • Demonstrates zero-shot cross-level skill transfer: skills learned on easier task families generalize to structurally harder unseen levels with no target-level experimentation.

Results

  • Claude Sonnet 4.6 baseline (ReAct): 2% success on hardest Interphyre level (catapult); HExA raises it to 77% (+75 pp).
  • Zero-shot transfer of skills from easier levels (down_to_earth, two_body_problem, pass_the_parcel) to catapult achieves 44% success with no target-level trajectories; cross-level transfer yields up to +36 pp from transferred hierarchical skills alone.
  • HExA outperforms ReAct and Reflexion baselines across all Interphyre levels.
  • Gains replicate across open-weight models (specific numbers in truncated text).

Limitations

  • Evaluated exclusively on 2D procedural physics simulation (Interphyre/PHYRE); generalization to other novel domains is asserted but not demonstrated here.
  • Experimentation is costly per instance (up to \(T\) tool calls per seed Ă— many seeds); amortization via skills reduces but does not eliminate this overhead.
  • Skill bank is capped (\(M_\text{max}\), \(N_\text{max}\)), requiring the evolver to prune; pruning quality depends on LLM judgment.
  • Cross-task transfer strips coordinate-level details; transfer fidelity degrades when target tasks require fine-grained quantitative grounding not captured by mechanism abstractions.
  • The paper acknowledges that even with HExA, frontier models still struggle—77% is the ceiling observed on the hardest level, leaving substantial headroom.

Relevance to Agentic AI / LLM Agents

HExA directly addresses one of the most fundamental open problems in LLM agents: adapting to genuinely novel environments without retraining. The actor–evolver–retriever architecture is a clean, generalizable pattern for training-free, in-context self-improvement that sidesteps catastrophic forgetting and closed-model constraints simultaneously. The hierarchical skill bank—where later skills build on earlier ones and mechanism-level abstractions transfer across task families—is a concrete mechanism for compositional knowledge accumulation, connecting to ongoing work on skill libraries (AutoSkill, SkillX, CoEvoSkills) but uniquely requiring no external feedback signal or ground-truth comparison. The Interphyre benchmark fills a gap for evaluating agents that must gather evidence rather than recall answers, which is a prerequisite for agents operating in open-ended physical or scientific domains.