Skip to content

Agentic Routing: The Harness-Native Data Flywheel

🕒 Published (v1): 2026-07-13 11:05 UTC · Source: Arxiv · link

Why this paper was selected

Positions routing as a harness-native data flywheel; directly actionable for agent harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Harness-native agentic routing treats model selection as a step-level decision conditioned on the full agent execution state (observations, tool history, verifier signals, recovery status), rather than on a static prompt. Every routing decision produces a structured arena record whose labels come from the environment, forming a self-improving data flywheel. Instantiated in OPENSQUILLA, a LightGBM cold-start ranker cuts costs ~90% on PinchBench and ~43% on DRACO versus a fixed strong-model baseline.

Problem

Existing model routing optimizes single-turn cost–quality trade-offs over static prompts, missing the stateful, multi-step, and recoverable nature of agent execution. A cheap model chosen at step \(t\) may trigger tool failures, context reconstruction, or verification recovery that raises total trajectory cost \(C(\tau)\) beyond what a stronger model would have cost; no existing router accounts for this because it cannot observe harness state \(h_t\). The decomposition Agent = Base Models + Harness demands that model selection be a joint optimization over a specialized model space and a specialized harness adaptation space.

Method

The paper formalizes routing as a state-conditioned operator \(g\) that selects a model set \(S_t = g(\mathcal{M} \mid h_t)\) at each execution step to optimize the Pareto front:

\[\min_g \left( \mathbb{E}_{q,\tau}[\ell(\tau)],\ \mathbb{E}_{q,\tau}[C(\tau)] \right), \quad S_t \subseteq \mathcal{M},\ 1 \le |S_t| \le K\]

Two regimes:

Singleton routing (\(k=1\)): fields one best-fit model using a per-turn reward \(\rho_t \in [0,1]\) (passing unit tests, tool-call success, verifier/judge scores) to densify supervision:

\[m_t = \arg\min_{m \in \mathcal{M}_t} \mathbb{E}_\tau[\ell(\tau) + \lambda C(\tau) - \beta\rho_t \mid S_t = \{m\}, h_t]\]

Multi-model ensemble routing (\(k>1\)): selects a proposer set \(P_t\) plus aggregation policy \(a_t \in \mathcal{G}\) (voting, verifier-based selection, judge models, fallback), with a complementarity regularizer \(-\alpha V(P \mid h_t)\) that rewards proposer sets with decorrelated failure modes.

OPENSQUILLA implements a four-layer stack: (1) token-complexity filter routing trivial requests to cheap models; (2) task-type classifier (coding/reasoning/chat/tool-use); (3) context-aware refiner incorporating harness state to prune \(\mathcal{M}\); (4) LightGBM ranker selecting cheapest capable model from remaining candidates. Arena records with schema \((q, h_t, S_t, z_t, c_t, y)\) accumulate to train progressively stronger routing policies. The LightGBM router is an open cold-start seed; the closed API accumulates arena records for later iterations.

Key Contributions

  • Formalizes harness-native agentic routing as step-level Pareto optimization over task loss and trajectory cost conditioned on full harness state, not just the user prompt.
  • Introduces the harness-native data flywheel: routing decisions are environment-labeled training records; better routers lower cost, enabling more traces, which improve routers further.
  • Proposes a complementarity regularizer \(V(P \mid h_t)\) for multi-model ensemble selection that rewards decorrelated failure modes, enabling cheaper model sets to outperform expensive singletons.
  • Releases OPENSQUILLA with a four-layer router (token-complexity filter → task classifier → context-aware refiner → LightGBM ranker) as an open cold-start seed.
  • Defines the arena record schema \((q, h_t, S_t, z_t, c_t, y)\) as a structured supervision interface between harness execution and router/model training.

Results

  • Singleton LightGBM router preserves task quality while cutting realized cost by ~90% on PinchBench versus a fixed strong-model baseline.
  • Singleton LightGBM router cuts realized cost by ~43% on DRACO versus the same baseline.
  • Multi-model ensemble routing pushes the DRACO quality frontier upward while often simultaneously lowering cost relative to single-model routing.
  • (Paper text was truncated before per-task breakdowns and ablations; no additional per-benchmark numbers are available in the provided text.)

Limitations

  • The LightGBM cold-start ranker is described explicitly as a seed, not a final policy; routing quality is expected to improve only after arena records accumulate—cold-start performance on novel model pools may be weak.
  • The complementarity regularizer \(V(P \mid h_t)\) is approximated via historical error decorrelation and disagreement surrogates; quality of this approximation in low-data regimes is uncharacterized.
  • Harness policies (tool schemas, context management, recovery) are fixed during evaluation; gains are attributed only to model allocation, leaving joint harness–router optimization unexplored.
  • The open LightGBM router and arena records are separated by a deliberate open-source boundary (later iterations served via closed API), limiting full reproducibility of the flywheel's later stages.
  • Evaluation covers two benchmarks (DRACO, PinchBench); generalization to other agentic settings is not demonstrated in the provided text.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a structural gap in how LLM agents use models: it replaces the static Agent = Model + Harness decomposition with a dynamic, state-conditioned capability allocation system, which is increasingly necessary as model specialization (coding, long-context, tool-use, reasoning) makes no single model Pareto-dominant. The data flywheel framing reframes every routing decision as agent-native training data, connecting harness engineering to model improvement in a feedback loop—a concrete mechanism for the "alternating progress" between base models and execution systems that is central to current agent scaling debates. For researchers tracking agentic AI, this work defines a formalism and open implementation for step-level model orchestration that goes well beyond cost-saving cascades, and the arena record schema offers a standardized interface for harness-native supervision that could generalize to multi-agent systems, tool-use fine-tuning, and verifier-guided RL.