Skip to content

Building Agent Harnesses for Scientific Curation from Multimodal Sources

🕒 Published (v1): 2026-06-19 00:36 UTC · Source: Arxiv · link

Why this paper was selected

Title literally 'Agent Harnesses'; multimodal scientific curation; strong MSR-adjacent group

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Beaver is an agent harness for extracting structured records from full scientific papers containing text, tables, and figures, outperforming frontier agents by over 23 absolute GRAS points. The paper demonstrates that harness design—not base model strength—is the primary determinant of performance on long-horizon, multimodal scientific curation tasks. The core insight is that staged task decomposition, multimodal evidence tooling, and artifact-grounded self-improvement loops together enable reliable schema-constrained extraction with provenance.

Problem

Frontier agents fail at scientific structured curation because: (1) evidence is scattered across long text, dense tables, and figures; (2) records require cross-modal reasoning and normalization rather than span copying; and (3) existing benchmarks (PubTables-1M, SciTSR, SciClaimEval) are already saturated by frontier models and don't require full-paper, rich-schema, expert-gold, long-horizon curation. The task demands auditability—each extracted value must be traceable to source evidence—which generic harnesses don't provide.

Method

Beaver wraps a frontier base agent (Copilot CLI + GPT-5.4) with three components:

Multimodal Evidence Tooling (\(U\)): Paper-to-markdown transformation (section-level files), parsed table viewer, and calibrated gridline overlay for plot reading. Tools produce derived artifacts that the agent reads rather than raw PDF/XML.

Task Scaffolding (\(\{\omega_k\}_{k=1}^K\)): Decomposes curation into \(K=4\) sequential stages (easier paper-level → harder arm/endpoint-level). Each stage \(k\) predicts a cumulative table: $\(\hat{Y}^{(k)} = h_k\!\left(p,\, A(p),\, \hat{Y}^{(k-1)};\, \omega_k, U\right)\)$ where \(A(p)\) are derived artifacts and \(\hat{Y}^{(0)} = \emptyset\). Within each stage the scaffold prescribes: preprocess → navigate → multimodal reasoning → extract with provenance → normalize → review/refine.

Artifact-Grounded Autoresearch: A two-level optimization loop (Algorithm 1) over \(T=3\) outer iterations × \(J=10\) inner steps = 34 total. Each outer step identifies the lowest-performing stage, initializes a workspace with logs/trajectories/provenance traces/evaluation outputs, proposes patches to \(\omega_k\), \(U\), or prompts \(\Pi\), reruns that stage on an 8-paper dev set, and accepts or discards the patch. The filesystem serves as persistent memory for grounded diagnosis.

Evaluation uses GRAS (Gold-Referenced Attribute Score): bipartite matching via Hungarian assignment between predicted and gold rows, then attribute-type-specific scoring (exact match, token-F1, relative accuracy for numerics), averaged over all gold rows with one-sided penalties for unmatched rows.

Key Contributions

  • Beaver harness: A task-specific agent harness combining multimodal tooling, staged scaffolding, and artifact-grounded self-improvement for scientific curation.
  • New benchmark: 23-paper evaluation corpus from PubMed/PMC on Alzheimer's disease clinical trials; 20 high-value attributes; 425 references, 58,058 curated rows; expert-gold reference data; first to require full-paper + multimodal + rich-schema + long-horizon curation simultaneously.
  • GRAS metric: Attribute-aware, gold-referenced aggregate with bipartite row alignment and one-sided missingness penalty.
  • Empirical finding: Harness design explains >23-point GRAS gap over same-backbone baselines; model swapping alone does not close this gap.
  • Ablation evidence: Each of the three components contributes independently (task scaffolding: −20.5 pts; multimodal tooling: −14.9 pts; provenance traces: −10.3 pts).

Results

  • Beaver: 81.0 ± 0.7 GRAS vs. best baseline Codex CLI (GPT-5.4): 58.6 ± 1.0 — +22.4 pts absolute.
  • vs. Copilot CLI (same base agent, GPT-5.4): 57.4 ± 1.4 — +23.6 pts, isolating harness contribution.
  • vs. Claude Code (Opus 4.7): 58.5 ± 2.6 — high-capability model swap does not help.
  • vs. Meta-Harness (GPT-5.4, SoTA on Terminal Bench 2.0): 48.0 ± 7.4 — generic harness optimization fails; Beaver beats it by 33 pts.
  • Largest per-attribute gains on endpoint.unit, change, percentchange, value, n.observed — attributes requiring cross-modal reasoning and normalization.
  • Ablation GRAS: no task scaffolding → 60.5 (−20.5); no multimodal tooling → 66.1 (−14.9); no provenance → 70.7 (−10.3).
  • Token budget: full Beaver uses 743M tokens over 34 autoresearch iterations; no-tooling variant uses only 555M (less compute, far worse quality).

Limitations

  • Evaluation set is small (23 papers, 8-paper dev set), restricted to Alzheimer's disease clinical trials on PMC Open Access.
  • Single disease area limits generalizability claims across domains.
  • Autoresearch budget (34 iterations, 743M tokens) is substantial; cost and compute are not analyzed for practical deployment.
  • Provenance traces are not included in GRAS evaluation—their contribution is assessed only qualitatively and via ablation.
  • The harness uses Copilot CLI as base; it's unclear how well the design transfers to other base agent interfaces.
  • Stage-localized diagnosis assumes failures are stage-separable; entangled cross-stage errors are not explicitly addressed.

Relevance to Harnesses / Meta-Harnesses

This paper is directly constitutive of the harnesses/meta-harnesses topic: it frames harness design as the central research object, introduces a concrete taxonomy of harness components (tooling, scaffolding, autoresearch loop), and provides the clearest empirical evidence to date that harness architecture dominates model selection for a well-defined agentic task. The artifact-grounded autoresearch loop (Algorithm 1) is itself a meta-harness: it is a harness that iteratively rewrites a task-harness using execution artifacts as feedback, a more structured and stage-localized variant of prior free-form self-improvement approaches like Meta-Harness. The 23+ point gap over Meta-Harness on the same backend demonstrates that domain-specific harness structure outperforms generic harness optimization, a key design principle for anyone building meta-harnesses targeting specialized pipelines.