Skip to content

ToFu: A White-Box, Token-Efficient Agent Harness for Researchers

🕒 Published (v1): 2026-07-13 11:26 UTC · Source: Arxiv · link

Why this paper was selected

White-box token-efficient harness design; directly targets researcher-buildable agent scaffolding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToFu is an open-source, MIT-licensed agentic harness that wraps LLMs with a six-module orchestration architecture to perform coding, file editing, and tool use. It achieves state-of-the-art results on SWE-bench Verified while consuming 28.4% fewer tokens on average than Claude Code. Its explicit design goal is to serve simultaneously as a practical research assistant and as a transparent, white-box research instrument for studying harness behavior.

Problem

Current agentic harnesses face four compounding problems: (i) excessive token consumption even for trivial queries, driving up cost; (ii) degraded non-English performance due to English-centric LLM pretraining; (iii) lack of local-deployment options for privacy-sensitive users; and (iv) a research-vs-product trade-off where benchmark-shaped open-source harnesses lack usable UIs, while polished commercial harnesses (e.g., Claude Code, Codex) use silently-updated proprietary models that are unsuitable as controlled research instruments. No existing harness resolves all four simultaneously.

Method

ToFu is organized into six modules: User Interface, Agent Orchestration Core (planner-worker-critic loop with checkpointing and a swarm scheduler), Model Abstraction Layer (provider-agnostic gateway), Capability Runtime (native tools + MCP extension adapters), State & Knowledge (persistent memory via BM25-ranked retrieval over structured Markdown records), and Context Management.

The token-efficiency mechanism is a three-layer context compaction pipeline applied at every reasoning step: 1. Size-aware budgeting: large tool outputs (search results, shell output) are externalized and replaced with compact previews and recoverable references; file reads are treated conservatively to avoid re-reads. 2. Cache-aware micro-compaction: older bulky tool outputs and stale reasoning blocks are replaced deterministically (no LLM call required) with concise placeholders, designed to avoid invalidating prompt-cache prefixes. 3. Query-aware semantic compaction (triggered near the context limit): a lightweight model (e.g., GPT-4o) produces a working-state summary that preserves user instructions, technical decisions, errors, and pending steps while discarding tangential history.

Multilingual mode wraps reasoning with a translate-then-reason workflow: user input is translated to English before the main model, and responses are translated back, with code blocks and marked spans protected from translation.

Swarm scheduler represents complex tasks as a DAG of agent objectives, executes independent sub-agents in parallel, injects prerequisite results into downstream agents, and supports reactive expansion where a master agent spawns additional agents based on intermediate evidence.

Memory retrieval uses BM25 over metadata and content (avoiding dense embedding latency), with on-demand loading rather than bulk injection.

Key Contributions

  • Open-source (MIT), white-box harness that simultaneously achieves leading benchmark performance and an application-level UI, closing the research-vs-product gap.
  • Three-layer context compaction yielding a 28.4% average token reduction (up to 43.6%) vs. Claude Code on SWE-bench Verified, with higher Pass@1.
  • Multilingual translate-then-reason mode; +2.5 accuracy points average on MAPS:SWE-bench across 10 languages.
  • Dependency-aware swarm scheduler enabling parallel sub-agent execution with DAG-based coordination.
  • MCP integration with Overleaf, GitHub, Notion, and Slack; supports calling external agentic systems (e.g., Claude Code) as tools.
  • BM25-based on-demand memory retrieval with project-scoped vs. global memory separation.

Results

All results on SWE-bench Verified (500 instances); baselines are Claude Code (commercial) and OpenCode (open-source CLI).

Pass@1 (higher is better): - Claude Opus 4.6 backbone: ToFu 83.2% vs. Claude Code 79.6% vs. OpenCode 74.2% - GLM 5.1 backbone: ToFu 80.4% vs. Claude Code 77.6% vs. OpenCode 71.6% - DeepSeek-v4-pro backbone: ToFu 80.2% vs. Claude Code 75.2% vs. OpenCode 71.8% - Average improvement over Claude Code: +3.8 pp; over OpenCode: +8.7 pp

Token efficiency (fewer is better): - ToFu vs. Claude Code: 663,881 vs. 837,764 avg. tokens/instance (Opus 4.6); 575,255 vs. 1,020,453 (GLM 5.1); 1,110,192 vs. 1,401,574 (DeepSeek-v4-pro) - Average token reduction: 28.4% (up to 43.6%)

Multilingual (MAPS:SWE-bench, Claude Opus 4.6): - Multilingual enhancement vs. direct source-language reasoning: +2.5 accuracy points average across 10 languages; improvement in 7 of 10 languages.

Notable finding: More token usage does not correlate with better performance; ToFu outperforms Claude Code with fewer tokens, suggesting substantial headroom in harness efficiency design.

Limitations

  • Evaluation is almost entirely on agentic coding (SWE-bench); broader research-assistant scenarios are assessed only via a small human preference study with three participants.
  • No systematic quantitative analysis of the token-efficiency vs. task-performance trade-off.
  • Multilingual enhancement has no improvement (or slight regression) in 3 of 10 tested languages.
  • The query-aware semantic compaction layer relies on a second LLM call (e.g., GPT-4o), introducing latency and additional cost not fully accounted for in the per-instance cost figures.

Relevance to Harnesses / Meta-Harnesses

ToFu directly targets the harness layer as both artifact and research object, making it a first-class contribution to harness design research. The three-layer compaction scheme operationalizes a principled answer to one of the key unresolved questions in harness engineering: how to manage context without degrading task performance — and demonstrates that more tokens is not a proxy for better task completion. The explicit white-box design and open-source release are positioned as an enabling substrate for harness-evolution research (cited: MetaHarness, HarnessForge, agentic harness engineering lines), providing a reproducible baseline with real product UX that prior research-oriented harnesses lacked. For researchers tracking this topic, ToFu is the most direct attempt to close the gap between the benchmark-harness and production-harness branches of the field.