Skip to content

OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning

🕒 Published (v1): 2025-02-16 21:18 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OctoTools is a training-free, extensible multi-agent framework that augments LLMs with heterogeneous external tools via standardized tool cards, a planner-executor agent pair, and a greedy toolset optimization algorithm. It achieves an average 9.3% accuracy gain over GPT-4o across 16 diverse reasoning benchmarks and outperforms AutoGen, GPT-Functions, and LangChain by up to 10.6% under identical tool and backbone conditions.

Problem

Existing tool-augmented LLM frameworks are either domain-specific, require fine-tuning or curated training data, cannot support multi-step reasoning across heterogeneous tools, or lack systematic benchmarking. No prior work provides a unified, training-free scaffold that is simultaneously extensible to new tools, evaluable across diverse domains, and controllable in which tools are activated per task.

Method

OctoTools structures tool-augmented reasoning as an iterated planner-executor loop over a trajectory \((s_0, s_1, \ldots, s_T)\):

  1. Tool Cards: Each tool \(d_i\) is wrapped in a structured card containing name, version, input/output schema, executable demonstrations, and user-authored metadata (limitations, best practices). Cards are modular; adding a tool requires only natural-language metadata and example commands—no framework changes.

  2. Planner Agent (four sub-roles): a Query Analyzer produces a high-level global plan; an Action Predictor generates a text action \(a_t\) (sub-goal + tool selection) at each step; a Context Verifier checks completeness and decides CONTINUE/STOP; a Solution Summarizer compiles the final answer from the trajectory.

  3. Executor Agent (two sub-roles): a Command Generator converts the planner's text action \(a_t\) into executable Python code \(o_t\); a Command Executor runs \(o_t\) in a Python environment and returns result \(r_t\), updating context \(s_t := (a_t, o_t, r_t)\).

  4. Toolset Optimization: A greedy \(O(n)\) algorithm evaluates each candidate tool \(d_i\) by computing \(\Delta_{d_i} = \text{Acc}(D_\text{base} \cup \{d_i\}) - \text{Acc}(D_\text{base})\) on a 100-example validation set and retains tools with \(\Delta_{d_i} > 0\), yielding \(D^* = D_\text{base} \cup \{d_i \mid \Delta_{d_i} > 0\}\).

Key Contributions

  • Standardized tool card schema that decouples tool specification from framework logic, enabling zero-shot tool integration without retraining.
  • Planner-executor separation: strategic text-based planning is decoupled from executable command generation, reducing compounding errors.
  • Task-specific toolset optimization: lightweight greedy search over \(n\) tools (versus \(O(2^n)\) brute force) to select beneficial subsets per domain.
  • Large-scale evaluation across 16 benchmarks spanning vision, math, science, medical, and agentic domains with controlled comparisons (same backbone, tools, and reasoning budget across frameworks).
  • Robustness analyses under weaker backbones (GPT-4o-mini, Qwen2.5 3B/7B) and injected tool failures.

Results

  • OctoTools (GPT-4o backbone, optimized toolset): 58.5% average accuracy across 16 tasks.
  • vs. GPT-4o zero-shot (49.2%): +9.3%
  • vs. GPT-4o CoT: +7.7%
  • vs. LangChain (51.2%): +7.3%
  • vs. AutoGen (50.8%): +7.7%
  • vs. GPT-Functions (51.0%): +7.5%
  • Full toolset (no optimization): 57.4% (+3.5% over base tool alone); optimized toolset: 58.9%.
  • GPT-4o-mini backbone: +7.1% average gain across 16 tasks vs. GPT-4o-mini baseline.
  • Qwen2.5 3B/7B backbones: +6.3–13.6% on representative tasks.
  • Performance scales monotonically with step budget; framework degrades gracefully under random tool failure injection.

Limitations

  • Toolset optimization requires a 100-example per-task validation set, which may not exist in zero-shot settings.
  • Greedy \(O(n)\) toolset search does not guarantee global optima; synergistic tool combinations may be missed.
  • Tool card authoring, while lightweight, still requires human-authored metadata and demonstrations per tool; partial LLM automation is proposed but not yet implemented.
  • Failure case analysis shows errors stem primarily from tool limitations (e.g., object-detector miscount) rather than the planning architecture—framework quality is bounded by tool quality.
  • Evaluation uses 200-example held-out test sets; statistical power may be limited for low-variance benchmarks.

Relevance to Harnesses / Meta-Harnesses

OctoTools is a canonical example of a meta-harness: it wraps heterogeneous external tools behind a uniform interface (tool cards) and orchestrates them via a reusable planner-executor control loop, exactly the pattern a digest or automation harness uses to wrap heterogeneous data sources and processing steps. The toolset optimization algorithm—greedily selecting which tools to activate per downstream task from a shared toolbox—is directly analogous to meta-harness logic that selects which sub-agents or pipeline stages to enable per topic or venue. The explicit separation of planning (what to do) from execution (how to call it) mirrors the harness pattern of separating orchestration logic from tool-specific adapters, and the standardized tool-card schema anticipates the need to add new tools (or in a digest context, new paper sources or LLM backends) with minimal framework surgery.