Skip to content

Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents

🕒 Published (v1): 2026-06-25 17:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Glite ARF is an open-source Python framework that wraps parallel LLM coding agents (Claude Code, Codex CLI) in a deterministic verifier layer, turning a research campaign into an auditable, reproducible pipeline. The central insight — "verifier-driven research" — is that rules governing research must live in code that fails loudly, not in prose agents are merely asked to follow. Applied to the BEA 2026 vocabulary-difficulty shared task, it achieved 1st place (closed track) and 2nd place (open track) across three languages with 273 tracked tasks and ~$498 total spend.

Problem

Naive delegation of empirical research to LLM coding agents does not scale beyond a handful of experiments: agents follow most instructions but the small fraction they skip compounds into fabricated citations, contaminated data splits, stale summaries, and out-of-scope edits. Prompt-encoded rules exhibit a persistent low-single-digit failure rate per invocation; over hundreds of tasks this produces dozens of broken artefacts. Existing experiment trackers (MLflow, W&B), versioning tools (DVC), and workflow engines (Snakemake) each address parts of this problem but none combines task-level worktree isolation, machine-checked artefact contracts, immutable corrections overlays, and comprehensive agent logging in a single repository-native framework designed for non-deterministic actors.

Method

ARF defines a three-role stack: (1) the human researcher selects hypotheses; (2) coding agents implement isolated tasks; (3) deterministic Python verificator scripts enforce structural rules before any artefact is committed or merged.

Seven structural principles are enforced by this verifier layer: 1. Task isolation — each task lives in its own folder and git worktree/branch; a pre-merge verifier (PM-E003) rejects any commit touching files outside that folder. 2. Immutability with corrections overlay — completed task folders are frozen; errors are repaired by downstream tasks writing correction files in their own corrections/ directory, applied at read time by aggregators. 3. Aggregators-only cross-task reading — all cross-task views are produced by scripts under arf/scripts/aggregators/; no agent ever walks tasks/ directly. 4. Materialised project overview — arf/scripts/overview/materialize.py regenerates a committed, GitHub-renderable dashboard; no human ever reads a manually maintained summary. 5. Spec-verified artefacts — every produced file carries a versioned spec_version field and is validated against a corresponding verificator script before commit; errors block merge. 6. Comprehensive logging — every CLI invocation is wrapped by run_with_logs.py; a verifier refuses to mark a step complete if the expected command log is missing. 7. Subagent isolation — complex tasks run as a chain of context-bounded subagents (research → planning → implementation → analysis → reporting), each seeing only the inputs it needs.

Parallelism is achieved by assigning each task a separate git worktree (task/<task_id>) and branch; the researcher opens one coding-agent session per task. Up to twelve sessions ran simultaneously on a single 48 GB Mac.

Key Contributions

  • Verifier-driven research paradigm: structural rules enforced by deterministic Python scripts rather than prose instructions, analogous to test-driven development.
  • Open-source ARF framework with seven structural principles, a versioned artefact specification system, and a materialised human-facing overview.
  • Immutable corrections overlay: enables post-hoc correction of merged artefacts without rewriting history — applied at aggregation time.
  • Demonstrated scalability: 273 tracked tasks, up to 12 parallel agents, across three multi-week campaigns in three distinct domains on identical framework code.
  • External, independently adjudicated benchmark anchor (BEA 2026 shared task) — leaderboard, test labels, and RMSE metric all outside authors' control.
  • Leakage detection via structured provenance: per-fold, per-feature audit trails allowed localisation of four target-leaking feature sets (correcting implausible RMSE 0.609 → 0.802) within minutes.
  • Overhead characterisation: ARF's own scripts constitute ~25% of commands but only ~1% of wall-clock time across campaigns.

Results

  • BEA 2026 closed track: 1st place on all three L1s (ES RMSE 0.903, DE 0.885, CN 0.776) vs. baseline (ES 1.257, DE 1.258, CN 1.140) — average 29.9% RMSE reduction.
  • BEA 2026 open track: 2nd place on all three L1s (ES 0.754, DE 0.764, CN 0.660) vs. baseline (ES 1.198, DE 1.166, CN 1.034) — average 35.9% RMSE reduction.
  • Best single component: LLaMA-3.1-8B LoRA regression head, K-fold RMSE 0.831 at 0.13% trainable parameters.
  • Campaign scale: 273 tracked tasks, 146 experiment runs, 129 feature sets, 1,161 feature columns; 12 peak concurrent agents; $498.31 total spend.
  • Isolation held at scale: across 305 merges in two mined external campaigns (WSD + Ace-CEFR), zero cross-task data corruptions and zero merge conflicts reached main.
  • WSD concurrency: sustained bursts peaking at 12 simultaneous tasks over 74 days; framework overhead ~1% of wall-clock time (Figure 5).
  • Compliance audit: per-column metadata enabled reclassification of 57/129 feature sets as closed-track-eligible after five rule clarifications; rebuilt system retained 1st place.

Limitations

  • Role 1 remains human-driven by design: hypothesis selection is explicitly not automated; authors state frontier models in mid-2026 do not reliably perform research-direction selection at multi-week campaign timescales.
  • Verifiers are structural, not semantic: they cannot detect a wrong analysis, an inappropriate baseline, or methodologically flawed experimental design — scientific merit judgement stays with the human.
  • Self-run campaign evaluation: two of three campaigns (WSD, Ace-CEFR) use author-held or author-run logs; only BEA has independent adjudication.
  • Single-team authorship of framework and campaign: the framework was designed in response to failures the same team encountered, which may bias which failure modes are addressed.
  • No ablation of individual principles: the paper does not isolate the contribution of each of the seven structural principles to the observed error-reduction outcomes.
  • Scale ceiling unstated: the paper does not characterise how the framework performs at concurrency levels beyond twelve agents or across significantly larger task graphs.

Relevance to Harnesses / Meta-Harnesses

Glite ARF is a canonical example of a research meta-harness: it does not implement the experimental logic itself but wraps arbitrary coding agents in a structural envelope that enforces reproducibility, provenance, and auditability at the campaign level. The verifier-driven paradigm — rules in deterministic code rather than prose, analogous to CI gates — is directly applicable to any harness that orchestrates multiple LLM agents over long-horizon pipelines, including paper-digest harnesses that must prevent stale summaries, contaminated outputs, or silent task failures. The corrections overlay (immutable history + read-time patching) and materialised overview patterns are immediately reusable design primitives for any harness that must maintain a coherent human-readable state across many parallel agent runs. The measured 1% wall-clock overhead for structural enforcement provides empirical evidence that harness-level safety machinery need not impose significant runtime costs.