Steerability via constraints: a substrate for scalable oversight of coding agents¶

🕒 Published (v1): 2026-07-02 16:24 UTC · Source: Arxiv · link

Why this paper was selected

Constraint-based steerability for coding agents; directly addresses scalable human oversight bottleneck

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper argues that classical software-engineering constraints (linters, type checkers, runtime validators, frozen dataclasses) are cheaper and more reliable than prompt-based scaffolding for overseeing coding agents. A 2×2 controlled experiment shows that combining a structured Python substrate with a 200-LoC docs CLI raises a small reviewer model's backdoor-detection recall from 54.5% to 90.9% on 11 stratified backdoors. The full system sketch wraps these substrate conventions in a VM sandbox, a network egress proxy, and an intercepting LLM inference proxy.

Problem¶

As coding agents produce thousands of lines autonomously, human (or trusted-model) oversight becomes the bottleneck: unconstrained codebases dilute reviewer attention, prompt-level steering is unreliable, and agentic scaffolding that saturates the context window actively harms verifiability. Existing tooling (Aider, MCP) optimizes the writer's context; nothing systematically structures the reviewer's context so that a small, fixed reviewer can audit code reliably.

Method¶

The core idea is steerability via constraints: every rule the agent must follow is enforced by a non-advisory tool (ruff with security/bugbear plugins, pyrefly, beartype, pydantic extra="forbid", jaxtyping-typed tensors, frozen/slotted dataclasses) rather than a prompt instruction. A pure core/ / impure runners/ split ensures function-local reasoning suffices. A 200-LoC docs CLI (built on griffe) implements a projection layer: docs module X renders public API + Invariants:/Conventions: blocks; docs symbol X.Y renders a single symbol's signature and docstring. The reviewer (Gemma 4 e4b, Q4_K_M quantization, 8B-class) is run in narrow per-function multi-pass mode—four focused passes per function: self-consistency, input/output contract, side-effects, and invariant compliance—sized so a single-shot full-codebase review would exceed the reviewer's effective context (\(\leq 4\text{K}\) tokens), forcing per-function review. The sandbox adds a qemu/KVM VM with virtiofs workspace sharing, a tinyproxy filtering egress to an allowlist, and a LiteLLM-style inference proxy that intercepts every agent↔model message for semantic probing.

Key Contributions¶

A start-to-end system sketch composed entirely of components under 100 LoC each (VM launcher, network proxy, inference proxy, docs CLI).
A concrete Python substrate convention set (Table 1) where every rule is tool-enforced, making contracts textually visible without prose.
A 2×2 controlled experiment (substrate × tools) on 11 stratified backdoors (L1 syntactic, L2 semantic, L3 multi-file, L4 in-module invariant, L5 global invariant) showing independent additive contributions from substrate and tooling.
The projection principle instantiated: a small CLI suffices as a repository view for reviewers; centralized ARCHITECTURE.md is inferior to module-local invariant docstrings (L5 backdoors caught only when the invariant is co-located with the code).
Full artifact release: synthetic codebase, 11 backdoors, docs CLI, reviewer harness, and evaluation pipeline.

Results¶

Setup	Recall	Precision	L1	L2	L3	L4	L5
C1: unconstrained, no tools	54.5%	100.0%	2/2	1/2	2/3	1/1	0/3
C2: unconstrained, docs CLI	81.8%	100.0%	2/2	1/2	3/3	1/1	2/3
C3: constrained, no tools	81.8%	88.9%	2/2	1/2	2/3	1/1	3/3
C4: constrained, docs CLI	90.9%	95.7%	2/2	1/2	3/3	1/1	3/3

Substrate and docs CLI contribute independently and monotonically.
Constraints alone (C3) reduce precision to 88.9%—the reviewer over-flags without a way to efficiently verify invariants; the docs CLI partially restores precision (95.7%).
L5 (global invariant) backdoors: 0/3 with no substrate or tools → 3/3 with constraints (module-local invariants); L3 (multi-file) backdoors: caught only when the docs CLI provides cross-module signatures.
An earlier single-pass protocol reversed the effect (4/8 C4 vs. 5/8 C1), demonstrating the multi-pass review protocol matters as much as the substrate.

Limitations¶

Dataset is synthetic and tiny: 11 backdoors, ~500 LoC, hand-crafted scoring rubric.
Single reviewer model (Gemma 4 e4b); L2 semantic backdoors remain at 1/2 in all conditions—authors conjecture a 26B–70B reviewer may close the gap.
Single seed; no prompt search (DSPy) or naturalistic backdoors from CVE corpora.
The system sketch (VM, proxies) is a work-in-progress prototype, empirically evaluated only for the substrate-plus-docs claim; the inference proxy semantic probes are proof-of-concept and not rigorously validated.
The comparison between C3 and constrained cells is asymmetric: centralized docs are shown only in the unconstrained condition, mirroring real PR-review behavior but not a fully controlled comparison.
Scope limited to Python; principles asserted to generalize to Rust/Lean but not demonstrated.

Relevance to Harnesses / Meta-Harnesses¶

The paper directly contributes a reviewer harness: a multi-pass, per-function review protocol that wraps a small fixed model with structured tool calls (docs CLI) and focused prompt lenses, analogous to how meta-harnesses decompose LLM tasks into focused sub-invocations. The inference proxy—intercepting every agent↔model message at a single editable boundary—is precisely a meta-harness intercept layer, enabling steering, logging, and semantic filtering without modifying the underlying model or agent. The substrate-level approach inverts the usual harness design question: rather than enriching the harness to compensate for an unstructured codebase, it structures the artifact so the minimal harness suffices. This is directly relevant to meta-harness designers evaluating when substrate-level enforcement can replace scaffolding complexity.