LLM Agents Making Agent Tools¶

🕒 Published (v1): 2025-02-17 11:44 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

TOOLMAKER is an agentic framework that autonomously converts scientific paper code repositories into LLM-compatible tools, eliminating the need for human developers to manually integrate research software. Given a GitHub URL and task description, it installs dependencies, explores the codebase, plans, implements, and self-corrects via a closed-loop debugging loop. It correctly implements 80% of 15 complex benchmark tasks, substantially outperforming OpenHands (the SWE-bench SOTA agent) adapted as a baseline.

Problem¶

LLM agents are bottlenecked by their fixed toolsets, which must be hand-crafted by developers before deployment. Prior tool-creation approaches (CRAFT, CREATOR, LATM) can only build simple, from-scratch tools because they lack OS interaction and cannot handle the complex dependency chains, model downloads, and multi-step orchestration typical of real scientific software. Researchers in healthcare and life sciences, where relevant code exists publicly but is technically demanding to deploy, are left without access.

Method¶

TOOLMAKER structures tool creation as a two-stage agentic workflow with a formal state model $s = (h, e) \in \mathcal{H} \times \mathcal{E}$, where $h$ is conversation history and $e$ is a Docker container environment state.

Stage 1 — Environment Setup: The INSTALL_REPOSITORY agent clones the repo, reads documentation, installs dependencies, and downloads model weights inside a python:3.12 Docker container. All write actions $\mathcal{A}_w$ are recorded and concatenated into a reproducible Dockerfile/bash script. A checkpoint snapshot $\bar{e}$ is saved.

Stage 2 — Tool Implementation: An EXPLORE agent gathers codebase context (on a clean history $h_\emptyset$ to avoid context pollution), then PLAN produces a step-by-step implementation plan, and IMPLEMENT generates a candidate Python function.

Closed-loop self-correction: The candidate tool is executed via RUN_IMPLEMENTATION; ASSESS_TOOL_OUTPUT judges whether output is plausible and error-free. On failure, DIAGNOSE_ERROR inspects intermediate files without resetting the environment, REIMPLEMENT produces a new candidate, and SUMMARISE compresses the attempt into a summary $\sigma$ appended to the restored conversation. The environment is reset to $\bar{e}$ each iteration. The loop runs until success or budget exhaustion.

The execution environment is sandboxed in Docker, with state rollback via Docker checkpointing and an in-container HTTP server exposing the action set $\mathcal{A}$.

Key Contributions¶

TOOLMAKER framework: first agentic system that converts existing scientific repositories (paper + code) into LLM-callable tools, with full OS interaction and dependency management.
TM-BENCH: a benchmark of 15 diverse scientific tasks (pathology, radiology, omics, 3D vision, NLP, tabular data) with 42 test invocations and 124 unit tests, evaluated in a fully open-ended environment (no pre-installed dependencies).
Formal workflow algebra: explicit state-space model distinguishing LLM calls ($\mathcal{H} \to \mathcal{H} \times \mathcal{M}$), environment interactions ($\mathcal{E} \to \mathcal{E} \times \mathcal{O}$), and agents ($\mathcal{H} \times \mathcal{E} \to \mathcal{H} \times \mathcal{E} \times \mathcal{R}$).
Closed-loop self-correction with history compression: per-iteration summaries prevent context explosion across debugging rounds.
Docker-based reproducibility: tools are packaged as portable Docker images executable on any machine.

Results¶

TOOLMAKER correctly implements 12/15 tasks (80%), passing all unit tests for each.
OpenHands (SOTA SWE-bench agent, adapted as baseline with GPT-4o) correctly implements only 4/15 tasks (27%), failing environment installation entirely on 7 tasks.
Per-task costs range from **$0.23 to $2.90**, averaging ~$0.85; action counts range from 10 to 35 per task.
Self-correction iterations range from 1 to 9 per task; token usage ranges from ~95K to ~1.79M per task.
OpenHands uses far fewer tokens (36K–250K) but fails on the majority of tasks, indicating insufficient depth rather than efficiency.
TOOLMAKER fails on 2 tasks: pathfinder_verify_biomarker (0/6 tests) and nnunet_train_model (0/4 tests), both also failed by OpenHands.

Limitations¶

Evaluated on only 15 tasks; generalization to the broader space of scientific repositories is unvalidated.
The closed-loop runs without a termination criterion other than success or implicit token budget; pathological repositories may loop indefinitely.
Relies on GPT-4o (gpt-4o-2024-08-06); LLM ablations are mentioned but not detailed in the provided text.
Docker checkpointing dependency is non-trivial infrastructure; portability to non-Linux or resource-constrained environments is unclear.
Tool definitions require a human-supplied task description and example invocation, so the process is not fully zero-shot.
TM-BENCH is small (15 tasks) and domain-skewed toward medical imaging, limiting coverage of other scientific disciplines.

Relevance to Harnesses / Meta-Harnesses¶

TOOLMAKER is directly a meta-harness pattern: it is an agentic workflow whose output is not a result but a reusable tool (a harness component) consumable by downstream agent orchestrators. The two-stage structure—environment setup then implementation with closed-loop correction—is itself a harness that wraps arbitrary external code into callable, sandboxed interfaces, exactly the dynamic toolset expansion that meta-harnesses require at runtime. The formal state model (workflow algebra of LLM calls, environment interactions, and agents) offers a principled template for composing sub-harnesses without context leakage between stages. For researchers tracking harness engineering, TOOLMAKER demonstrates how the tool-creation layer can be automated and decoupled from the tool-use layer, a key architectural separation for building extensible agent meta-harnesses.