Compiling Deterministic Structure into SLM Harnesses¶

🕒 Published (v1): 2026-04-19 14:04 UTC · Source: Arxiv · link

Why this paper was selected

Compiles deterministic structure into SLM harnesses via semantic gradient descent

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SGDe (Semantic Gradient Descent) is a teacher-student framework that compiles agentic workflows for small language models (SLMs) into a fixed execution plan $\theta = \{G, P, C\}$—a DAG topology, system prompts, and deterministic code—entirely offline. A frontier teacher iteratively rewrites the harness by treating natural-language critiques as discrete gradients. On GSM-Hard-derived benchmarks, the compiled harness achieves 91.3% accuracy at $m=5$ training examples and 99.3% at $m=3$, with zero frontier-API calls at runtime.

Problem¶

SLMs (sub-billion to ~10B parameters) suffer from epistemic asymmetry: they cannot reliably self-correct because the same model cannot both execute and judge its own reasoning. Existing approaches either transfer behavior via continuous weight updates (distillation), or optimize prompts and DAG topology while keeping every node an LLM call (DSPy, TextGrad, AFlow)—so-called intra-substrate optimizers. PAL/PoT cross the substrate boundary but apply code offloading as a static whole-problem template rather than a per-node, trace-driven decision. No prior method optimizes which computations the SLM is responsible for at each individual node.

Method¶

SGDe optimizes the execution plan $\theta = \{G, P, C\}$ where $G=(V,E)$ is a DAG, $P$ is a set of per-node prompts, and $C$ is deterministic executable code. The teacher operates in a critic/optimiser split:

Forward pass: The student $S$ executes $\theta_t$ on a training batch $Z = \{(\tau_i, y_i^*)\}_{i=1}^m$, producing outputs $\hat{y}_{t,i}$ and traces $\mathrm{Tr}_{t,i}$. Empirical risk is: $$\hat{R}(\theta_t) = \frac{1}{m}\sum_{i=1}^m L(S(\tau_i;\theta_t), y_i^*, \mathrm{Tr}_{t,i})$$

Backward pass (semantic gradient): When risk exceeds threshold $\epsilon$, the critic generates a natural-language semantic gradient: $$g_\mathrm{sem} = T_\mathrm{critic}(\Phi, \mathrm{Tr}_t, \hat{R}(\theta_t))$$

Candidate generation: The optimiser proposes a revised plan: $$\theta_{t+1} = T_\mathrm{optimiser}(\Phi, g_\mathrm{sem}, \theta_t)$$ accepted only if $\hat{R}(\theta_{t+1}) < \hat{R}(\theta_t)$ (greedy hill-climbing).

Per-node, the teacher selects among three mutually exclusive substrate actions: 1. Prompt refinement — rewrite $p \in P$ for instruction quality failures. 2. Capability offloading — retype node $v$ as $c \in C$ (deterministic Python) when the SLM reliably fails at that computation (e.g., arithmetic). 3. Structural consensus — wrap $v$ in a fan-out/fan-in subgraph $G_\mathrm{cons}$ with $n$ parallel perturbed instances aggregated by deterministic majority vote, for high-variance steps the model is capable of but unreliable on.

Granularity operations (node decomposition/fusion) reshape the subtask partition without changing substrate. A PAC bound is derived: the teacher's prior restricts the effective hypothesis space to $|\Theta_T| = O(3^k)$ for $k$ distinct subtask types, yielding sample complexity $m \geq \frac{1}{\epsilon}(k\ln 3 + \ln\frac{1}{\delta})$, linear in $k$.

Key Contributions¶

Substrate compilation as a new optimization unit: per-node, trace-driven selection of LLM-call vs. deterministic-code vs. consensus-subgraph, replacing the static offload decision of PAL/PoT.
PAC-bounded convergence analysis: effective hypothesis space $|\Theta_T| = O(3^k)$; at $\epsilon=\delta=0.1$, sample complexity $m \gtrsim 11k + 23$, explaining empirical convergence from $m=3$.
Teacher-student harness compiler: frontier teacher used only offline; compiled harness runs zero frontier-API calls at inference time.
Structural consensus mechanism: deterministic fan-out/fan-in aggregation for variance-sensitive nodes, complementary to capability offloading.

Results¶

SGDe at $m=5$: $\mu=91.3\%$ (range 80.0–100.0%) vs. DSPy $\mu=65.0\%$ (+26.3% absolute) and zero-shot baseline $\mu=48.3\%$ on GSM-Hard-derived test set.
SGDe at $m=3$: $\mu=99.3\%$ (+34.3% over DSPy); non-monotonic in $m$ — batch composition matters more than batch size at small $m$.
SGDe at $m=10$: $\mu=95.0\%$ (stabilizes).
DSPy ceiling: max 73.3% across all $m$; SGDe worst case (80.0%) exceeds DSPy maximum.
Every winning harness combined capability offloading and structural consensus; neither mechanism alone ever produced a winner.
Heterogeneous task (boundary condition): SGDe $\mu=48.7\%$, DSPy $\mu=47.3\%$, baseline $\mu=45.3\%$—both methods collapse to near-baseline when task family is structurally mixed (large $k$), confirming PAC prediction.
Student: Qwen-2.5-1.5B (heavily quantized); teacher: Kimi 2.5 (offline only); $n=3$ runs per configuration.

Limitations¶

Empirical validation restricted to GSM-Hard-style structured-reasoning tasks; enterprise document extraction scenarios are motivation only, not validated.
Non-monotonic accuracy in $m$ (99.3% at $m=3$, 91.3% at $m=5$) indicates sensitivity to batch composition in the small-sample regime.
Performance degrades to near-baseline on structurally heterogeneous task mixtures; mixture-of-topologies routing is proposed but not implemented.
PAC bound ($m \gtrsim 11k+23$) is looser than observed empirical convergence—the teacher's strong prior accounts for the gap, but this prior is not formally characterized.
Requires a capable frontier teacher (Kimi 2.5 with long-context reasoning) to ingest multi-step traces and emit valid JSON DAGs; weaker teachers may not converge.

Relevance to Harnesses / Meta-Harnesses¶

SGDe is a direct instantiation of a meta-harness: a system whose output is itself an optimized harness $\theta = \{G, P, C\}$, not a task answer. The compilation loop (teacher rewrites the DAG, prompts, and code; student executes against training examples) is precisely the meta-harness pattern of programmatically generating and refining agent execution structures. The paper extends prior harness-engineering work (Lopopolo 2026, Trivedy 2026) by formalizing the compile-time/runtime split and adding PAC-theoretic convergence guarantees, making it directly relevant to anyone designing systems that automatically construct or improve agent harnesses rather than hand-authoring them. The three-way substrate action space (prompt/code/consensus) provides a concrete, principled vocabulary for harness node classification that future meta-harness frameworks could adopt.