Skip to content

TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents

🕒 Published (v1): 2025-10-08 02:18 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TinyScientist is a modular, open-source framework for building LLM-based research agents that decomposes the end-to-end research pipeline into four core stages (think, code, write, review) with built-in interactivity, extensibility via MCP, and budget/safety controllability. It ships as a Python package, web UI, and PyPI release, and matches or exceeds Agent Laboratory in paper generation quality. The key design thesis is that accessibility and modularity should be first-class concerns, not afterthoughts.

Problem

Existing auto-research frameworks (AI-Scientist, Agent Laboratory, AI co-scientist) are difficult to extend, lack meaningful human-in-the-loop interaction, and provide no budget or safety guardrails. Three concrete failure modes are identified: (1) opaque agent state prevents researchers from steering in real time; (2) tool integration is tightly coupled to framework internals, requiring deep code changes to add capabilities; (3) no cost ceilings or ethical filters, risking runaway spending or harmful outputs.

Method

TinyScientist organizes the research workflow as four composable, iteratively self-refining agents—Thinker, Coder (wrapping Aider), Writer, and Reviewer—all sharing a common iterative-refinement loop with a user-configurable reflection cap. Three cross-cutting feature components sit beneath every workflow stage:

  • Formatter: Converts intermediate outputs into structured tables that users can directly edit cell-by-cell, enabling precise intent injection between stages.
  • MCPClient: Wraps all external tools (code searchers, plot drawers, paper retrievers via Semantic Scholar) behind the Model Context Protocol, so new tools require only MCP config changes rather than code rewrites.
  • Checker: At each stage, enforces (a) hard budget ceilings by estimating per-stage cost ratios and dynamically reducing reflection steps or terminating early; (b) safety filters that block or flag outputs violating ethical constraints.

The Thinker samples \(n\) ideas, each refined over \(k\) iterations, scored on impact, feasibility, and novelty. The Writer follows a three-step protocol: draft → LaTeX error correction → citation insertion via Semantic Scholar API.

Key Contributions

  • Four-stage modular research pipeline (think → code → write → review) with shared iterative-refinement structure.
  • MCP-based tool integration replacing bespoke API wrappers, enabling zero-code tool swaps.
  • Tabular human-agent communication interface for stage-level and within-stage intent editing.
  • Checker module providing simultaneous budget control (dynamic reflection-step scaling, early termination) and safety filtering.
  • Open-source codebase, interactive web demo, and PyPI package (tiny-scientist) for broad accessibility.

Results

  • Writing quality (ML domain): TinyScientist 4.06 vs. Agent Laboratory 3.96 (LLM eval, 5-point scale); 3.92 vs. 3.92 (human eval).
  • Idea quality (ML domain): 3.82 vs. 3.74 (LLM); 3.85 vs. 3.83 (human).
  • Writing quality (biology, OOD): TinyScientist 3.84 vs. AgentLab 3.41 (+0.43 LLM); 3.73 vs. 3.55 (human) — largest improvement attributed to richer tool use.
  • Tool ablation (ML writing): +0.13 writing, +0.04 idea (LLM); +0.13 writing, +0.15 idea (human) when MCP tools enabled vs. disabled.
  • Safety: 18/20 unsafe biology tasks blocked at Thinker stage; remaining 2 flagged at later stages.
  • All comparisons use gpt-4o-mini backbone for both systems; 20 ML tasks (in-distribution) and 20 biology tasks (OOD, from SciSafetyBench).

Limitations

  • Quality gains over Agent Laboratory are marginal in the ML domain; the framework's added complexity may not be justified for ML-only use.
  • Prompt design is ML-centric, causing degraded OOD performance on biological tasks—both frameworks score lower there.
  • Coder stage relies on an external tool (Aider), introducing a dependency not controlled by the framework.
  • Budget estimation relies on heuristic per-stage cost ratios; the paper acknowledges estimation errors can still lead to overspending before early termination triggers.
  • No evaluation of latency, wall-clock cost, or failure rate of the full pipeline under real research conditions.
  • Safety evaluation uses a single benchmark (SciSafetyBench) with only 20 tasks; generalization of safety claims is not established.

Relevance to Harnesses / Meta-Harnesses

TinyScientist is a direct example of a research meta-harness: it orchestrates heterogeneous sub-agents (Thinker, Coder, Writer, Reviewer) under a unified engine that manages control flow, tool dispatch, budget, and safety—exactly the concerns that define a meta-harness layer. The MCP-based tool integration pattern is directly applicable to any harness that needs to swap or extend capabilities without re-architecting the orchestration layer. The Checker's dynamic parameter adjustment (scaling reflection steps to fit budget) is a concrete implementation of runtime resource governance, a recurring design challenge in harness systems. The tabular interaction model also demonstrates one approach to human-in-the-loop control points, which meta-harnesses for research or automation pipelines must address.