From Model Scaling to System Scaling: Scaling the Harness in Agentic AI

🕒 Published (v1): 2026-05-25 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Theoretical treatment of system scaling vs model scaling; defines harness architecture requirements

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This position/framework paper argues that the next bottleneck in agentic AI is system scaling — improving the structured execution layer (the "harness") around foundation models — rather than further model scaling alone. It proposes a six-component decomposition \(P_H = \Phi(R, M, C, S, O, G)\) and identifies context governance, trustworthy memory, and dynamic skill routing as the three primary engineering bottlenecks. A Python reference harness, CheetahClaws, is released alongside comparative analysis of Claude Code and OpenClaw.

Problem

Agent evaluation remains model-centric: benchmarks collapse performance to final-task success or accuracy while treating memory, retrieval, tool use, orchestration, verification, and governance as secondary implementation details. This framing is inadequate because long-horizon agent behavior emerges from the interaction among all system components, not from the foundation model alone. Empirically, redesigning the agent–computer interface without changing the model can substantially move SWE-bench accuracy, and agents that appear strong under single-shot pass rates collapse under \(\hat{p}^k\) (probability of succeeding on \(k\) independent rollouts), revealing a reliability gap that endpoint accuracy conceals.

Method

The paper introduces a system-scaling framework that treats the agent harness as a first-class object of design. Agent performance over horizon \(H\) is formalized as:

\[P_H = \Phi(R, M, C, S, O, G)\]

where \(R\) = base reasoning quality (improved by model scaling), and \(M, C, S, O, G\) = memory quality, context-construction quality, skill-routing quality, orchestration quality, and governance quality (all targets of system scaling). Two components are further factorized:

\[M = (\text{precision, durability, retrievability, verifiability})$$ $$C = (\text{relevance, compactness, traceability, refresh policy})\]

Three bottlenecks are analyzed against this framework:

• Context governance (C): Treats context assembly as a selection policy rather than a buffer — weighting semantic relevance, penalizing verbosity, preferring recently validated content, and recording provenance. The threat is "exposure without access": token count grows but attention dilutes to non-relevant material.
• Trustworthy memory (M): Makes trust a runtime decision: retrieval scores combine staleness penalty (time since last verification) and a confidence-gated risk term, treating recalled content as a hypothesis until re-checked against the live environment. The threat is "stale-but-confident" memory acting on invalidated facts.
• Dynamic skill routing (S): Requires four conditions — specificity (capability scope), selectivity (invoke the right skill), composability (post-conditions chain), and verifiability (explicit post-condition checks). The threat is "confident-but-unchecked" subagent output.

Prompt, skill, and memory are further characterized as temporal layers: prompt for local control, skill for task-level reuse, memory for longitudinal persistence.

A reference harness, CheetahClaws (Python), is built and compared against Claude Code (TypeScript, vendor) and OpenClaw (TypeScript, community) along axes of context governance, memory representation, and skill routing. CheetahClaws stores per-entry confidence and recency as first-class fields used directly in retrieval ranking and conflict resolution; the other two derive trust implicitly from access patterns.

Key Contributions

• System-scaling framing: Reframes agentic AI progress as a harness-engineering problem; model scaling improves \(R\), while system scaling improves \(M, C, S, O, G\).
• Six-component harness framework: Formal decomposition \(P_H = \Phi(R, M, C, S, O, G)\) with sub-axis factorizations for \(M\) and \(C\); each axis is an independently tunable engineering lever.
• Three-bottleneck analysis: Concrete characterization of context governance, trustworthy memory, and dynamic skill routing including dominant failure modes and system moves for each.
• Evaluation agenda: Proposes process-level and longitudinal benchmarks measuring trajectory quality, memory hygiene, context efficiency, communication fidelity, verification cost, and safe evolution — beyond one-shot success.
• CheetahClaws reference harness: Open-source Python harness used as a concrete comparison point; released at github.com/SafeRL-Lab/cheetahclaws.

Results

This is a position/framework paper with no original empirical benchmark evaluations. Results cited from prior work to support claims: - SWE-agent: redesigning the agent–computer interface while holding the backbone model fixed substantially improves SWE-bench accuracy (establishing harness as a separable performance lever). - τ-bench: agents with strong single-shot pass rates collapse under \(\hat{p}^k\), exposing a reliability gap hidden by endpoint accuracy. - Anthropic multi-agent studies: orchestrator+subagent configurations produce substantial gains on breadth-first research tasks but introduce coordination failures single-agent metrics miss. - Field-level benchmark analysis: many agent results are non-Pareto-optimal once prompting strategy, costs, and demonstrations are controlled (suggesting reported scores are model-plus-harness scores, not model scores).

Limitations

• \(\Phi\) in Equation 1 has no closed form; the six factors are not strictly orthogonal, making the decomposition conceptual rather than quantitative.
• \(R\) and \(C\) are the hardest to separate empirically: a stronger reasoning substrate can compensate for noisier context and vice versa.
• No ablation or controlled empirical comparison of CheetahClaws, Claude Code, and OpenClaw on matched tasks; the comparison is design-pattern-level, not performance-level.
• The manuscript is explicitly under active development; claims are presented as a research agenda rather than settled findings.
• The evaluation agenda is proposed but not instantiated with implemented benchmarks measuring the proposed process/longitudinal metrics.

Relevance to Harnesses / Meta-Harnesses

This paper is the most directly relevant possible work for someone tracking this topic: it coins "scaling the harness" as a formal research program and provides the field's first systematic decomposition of what a harness is (\(R, M, C, S, O, G\)) and what its engineering bottlenecks are. The three-bottleneck analysis (context governance, trustworthy memory, skill routing) gives concrete vocabulary and design axes for comparing harness architectures — directly applicable to evaluating or building meta-harnesses that coordinate multiple sub-harnesses. The temporal-layer view (prompt/skill/memory) maps well onto how meta-harnesses must manage state across varying timescales, and the CheetahClaws vs. Claude Code vs. OpenClaw comparison instantiates harness design trade-offs as reproducible reference points.