Skip to content

FARS: A Fully Automated Research System Deployed at Scale

🕒 Published (v1): 2026-06-30 13:30 UTC · Source: Arxiv · link

Why this paper was selected

Fully automated research deployed at scale; shows AI Scientist beyond curated demos

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FARS is a fully automated, multi-agent research system that autonomously runs the complete scientific workflow—from ideation through planning, experimentation, and paper writing—without human intervention during execution. In its first public deployment, FARS produced 166 complete AI/ML papers across 67 topics, evaluated by 282 structured human reviews. The deployment exposes the output distribution of a continuously running system, including weak outputs, negative results, and failure modes, rather than a curated sample.

Problem

Prior autonomous research systems (The AI Scientist, DeepScientist, AI-Researcher, etc.) are evaluated on selected demonstrations, human-framed topics, narrow benchmark tasks, or small predefined frontier goals. It remains unknown how such systems behave when deployed continuously at scale—autonomously selecting topics, executing full research pipelines, and generating large volumes of heterogeneous outputs without human scaffolding.

Method

FARS is a four-stage pipeline with stage-specific agents coordinated through a shared, auditable workspace:

  1. Ideation: Four specialized agents (Lead, Summarization, Peer Discussion, Evaluation) perform literature review, direction exploration, adversarial pre-filtering, proposal drafting, and iterative review-and-revision. Proposals must clear an automated quality gate before proceeding.

  2. Planning: Converts the accepted proposal into a machine-readable experiment contract—a two-level hierarchical JSON structure decomposing the study into ordered items across five task categories (environment configuration, baseline experiments, main experiments, effectiveness evaluation, analysis experiments). Automated validation enforces format, required fields, and ordering constraints.

  3. Experiment: An orchestration harness separates orchestration from execution, advancing one plan item at a time. Within each item: an experiment agent produces an execution plan, an independent review agent checks fidelity to the proposal, then the agent executes with full tool permissions. Dual verification (semantic self-assessment + deterministic harness checks) gates result recording. An effectiveness evaluation gates transition from main experiments to analysis. Infrastructure includes a layered environment with pre-configured ML dependencies, a curated 19-category skill library loaded on demand, high-abstraction GPU tool interfaces preventing uncontrolled resource allocation, checkpoint-based fault tolerance, and a unified model proxy for LLM/embedding/image APIs. The cluster comprises 160 NVIDIA GPUs.

  4. Writing: An Analysis Agent audits upstream artifacts and constructs a paper blueprint linking each claim to supporting experiments, figures, and citations. A Writing Agent then drafts the LaTeX manuscript section-by-section, with numerical verification against source artifacts during refinement to reduce hallucinated results.

The system was seeded with 9 AI research directions and ran for 417 hours over its first public deployment.

Key Contributions

  • End-to-end FARS architecture: Ideation → Planning → Experiment → Writing agents coordinated via a shared auditable workspace, fully autonomous during execution.
  • First large-scale public deployment of an idea-to-paper AI research system: 166 complete papers spanning 67 fine-grained AI/ML topics, with preserved intermediate artifacts (hypotheses, plans, code, logs, results, manuscripts).
  • Systematic human evaluation corpus: 282 structured reviews covering 140 papers, including overall ratings, soundness/presentation/contribution sub-scores, integrity checks, and LLM-use disclosure.
  • Characterization of recurring failure modes at deployment scale: narrow experimental scope, methodological limitations, and integrity issues.

Results

  • Scale: 166 complete papers produced across 67 fine-grained AI/ML topics; 417 deployment hours; 21.6 billion model tokens consumed; ~\(186,000 total cost; ~\)1,120 per paper amortized.
  • Human review coverage: 282 structured reviews across 140 of 166 papers (26 papers received no reviews).
  • Quality signal: Reviews indicate FARS can produce review-worthy and occasionally strong AI/ML research artifacts at public-deployment scale.
  • Recurring weaknesses identified by reviewers: narrow experimental scope, methodological limitations, and integrity issues (specific score distributions are not quoted in the provided text).
  • No head-to-head numeric comparison against prior systems is reported in the provided excerpt; prior systems are compared qualitatively on scope and evaluation methodology.

Limitations

  • Experimental scope is frequently narrow—a recurring failure mode flagged by human reviewers.
  • Methodological limitations (e.g., insufficient baselines, shallow ablations) appear systematically.
  • Integrity issues are present in a non-trivial fraction of outputs (exact fraction not given in provided text).
  • The ideation–execution gap: proposals may be rated acceptable yet downstream execution may fail to fully realize the hypothesis.
  • Evaluation relies on volunteer reviewers whose coverage is incomplete (26/166 papers unreviewed) and who may themselves use LLM assistance.
  • Current deployment is restricted to AI/ML topics where hypotheses can be operationalized as executable code against shared benchmarks; generalization to other scientific domains is undemonstrated.
  • The Writing stage uses a proprietary diagram synthesis model (NanoBanana-Pro), introducing an external dependency not independently validated.
  • Cost (~$1,120/paper) and compute requirements (160 GPUs) limit accessibility.

Relevance to Agentic AI / LLM Agents

FARS is a landmark real-world deployment demonstrating that multi-agent LLM systems can autonomously execute long-horizon, multi-day research workflows at scale—a capability regime well beyond standard agentic benchmarks. The architecture directly addresses known LLM agent failure modes (single-pass task collapse, premature completion) through hierarchical planning contracts, orchestrator/executor separation, and staged effectiveness gates, offering concrete design patterns for robust long-horizon agentic pipelines. The auditable artifact-centered workspace and checkpoint-based fault tolerance are transferable infrastructure principles for any agentic system that must operate unattended. The documented failure modes (scope narrowing, integrity drift, hallucinated results) provide an empirically grounded picture of where current LLM agents systematically break down under real autonomous deployment conditions.