Skip to content

BrainPilot: Automating Brain Discovery with Agentic Research

🕒 Published (v1): 2026-07-16 14:49 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BrainPilot is an open-source multi-agent harness for brain science research in which a PI agent orchestrates six specialist agents (Librarian, Experimentalist, Engineer, Writer, Auditor) grounded in a curated 7,233-item knowledge base and a 72-skill methodology library. All intermediate steps are recorded in a structured Graph of Trace to enable expert inspection and fabrication auditing. On Agents' Last Exam brain science tasks, a BrainPilot-GLM-5.2 configuration matches or approaches the performance of Codex (GPT-5.5) at roughly 57× lower cost ($0.08 vs. $4.66).

Problem

Multi-step brain science research demands coordinated literature retrieval, experimental design, data analysis, and interpretation across heterogeneous modalities, yet existing scientific agents lack domain grounding specific to neuroscience, are prone to hallucination and instruction drift in long workflows, and provide no defined intervention points for expert oversight—failures that are especially costly when agent outputs feed directly into scientific claims.

Method

BrainPilot implements a hierarchical multi-agent harness built on the PI Agent framework. The PI agent decomposes user goals and delegates to five specialists: Librarian (RAG-based literature synthesis), Experimentalist (protocol and control design), Engineer (code execution and reproducibility), Writer (document assembly), and Auditor (claim verification). Domain grounding comes from two offline-constructed assets: (1) a vector index over 7,233 neuroscience textbooks and papers, embedded with BAAI/bge-m3 and reranked with a cross-encoder (BAAI/bge-reranker-v2-m3) at query time; (2) a 72-skill library grouped into seven research domains, accessed through a two-stage skill router that separates skill discovery (keyword search returning candidates) from skill loading (full instructions fetched on demand) to limit context overhead. Every agent action, tool call, evidence pointer, and generated claim is written into the Graph of Trace, a structured auditable execution record. The Auditor independently classifies all numerical, artifact, and citation claims in delivered drafts as confirmed, unverified, or disputed against session-produced evidence, then saves a timestamped audit report. Three meta-skills (paper-to-skill, repo-to-skill, workflow-skill-creator) allow the skill library itself to be extended from new publications or repositories.

Key Contributions

  • Multi-agent harness with role specialization: six-agent team (PI + five specialists) with a shared system constitution, role definitions, messaging contract, and self-recording contract.
  • Curated domain-knowledge assets: 7,233-item neuroscience RAG corpus (11 disciplinary groups) and a 72-skill library derived from LLM authoring, paper distillation, and established analysis tools (DeepLabCut, MNE-Python, fMRIPrep, pycortex, SpikeInterface, netneurotools).
  • Graph of Trace: structured, auditable execution record linking subgoals, tool use, evidence, and claims across a full research session.
  • Integrated fabrication auditing: Auditor agent that verifies claims within the workflow rather than as a post-hoc step.
  • BrainPilotBench-v0: new benchmark with four brain science research tasks (preliminary version); also evaluated on three Agents' Last Exam brain science questions.
  • Meta-skills for library extension: paper-to-skill, repo-to-skill, and workflow-skill-creator enable structured addition of new methodology units with an explicit verification step.

Results

(Note: the paper text is truncated; only figures visible in the provided excerpt are reported.)

  • On the three Agents' Last Exam brain science tasks (scores appear to be per-task binary/partial credit):
  • BrainPilot (GLM-5.2): scores 1.00 / 0.70 / 0.09 at $0.08 total cost.
  • BrainPilot (GPT-5.5): 1.00 / 0.70 / 0.00 at $2.63.
  • BrainPilot (Opus-4.8): 1.00 / 0.70 / 0.00 at $1.02.
  • Codex (GPT-5.5), the stated SOTA baseline: 1.00 / 1.00 / 0.10 at $4.66.
  • Claude Code (DeepSeek-V4-Pro): 0.00 / 0.70 / 0.18 at $0.21.
  • Claude Code (GLM-5.2): 1.00 / 0.70 / 0.07 at $0.08.
  • BrainPilot (GLM-5.2) matches Codex on two of three tasks while costing ~57× less.
  • Full BrainPilotBench-v0 results and five end-to-end case studies are described but not included in the provided text.

Limitations

  • BrainPilotBench-v0 is preliminary (four tasks); coverage of the full brain science research space is narrow.
  • The 7,233-item knowledge corpus has a heavy psychiatric bias (30% psychiatry) and may underrepresent some subfields.
  • Skill library (72 skills) is manually curated; coverage gaps in emerging methods are likely.
  • Auditor checks claims only against session-produced evidence, not external ground truth—fabrications that are internally consistent can pass.
  • Domain grounding relies on an offline-built corpus; knowledge freshness requires corpus reconstruction.
  • The paper text is truncated; quantitative ablation results for the knowledge base, Graph of Trace, and Auditor components are not available in the provided excerpt.

Relevance to Harnesses / Meta-Harnesses

BrainPilot is a domain-specialized agent harness whose architecture maps cleanly onto harness design patterns: a coordinator agent dispatching to specialized sub-agents, a structured execution trace enabling replay and inspection (Graph of Trace as an analog to workflow logs), and a skill router that decouples skill discovery from skill loading to control context overhead. The three meta-skills (paper-to-skill, repo-to-skill, workflow-skill-creator) make the skill library itself extensible through a harness-mediated authoring pipeline—a concrete example of a meta-harness pattern where the harness governs its own capability expansion. For researchers tracking harnesses and meta-harnesses, BrainPilot demonstrates how structured execution traces, modular skill routing, and integrated verification stages can be composed into a domain-grounded harness while remaining framework-agnostic (built on the PI Agent framework, evaluated with multiple backbone LLMs).