Prompt-to-Paper: Agentic AI System for Bioinformatics¶

🕒 Published (v1): 2026-07-05 21:30 UTC · Source: Arxiv · link

Why this paper was selected

End-to-end agentic bioinformatics manuscript system with verifiable grounding — domain harness case study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Prompt-to-Paper (RLEv4) is a five-stage multi-agent pipeline that converts a research topic into a submission-formatted bioinformatics manuscript by chaining deterministic RAG retrieval, real autonomous experiment execution, and an eight-dimensional iterative quality scorer. A context-rich improvement loop with deep research cycles every ten iterations raises manuscript quality by an average of +17.96 points on a 0–100 scale across five case studies. End-to-end cost is approximately $0.31 per paper.

Problem¶

Existing automated manuscript generation systems share three critical failures: (i) claims are not grounded in verifiable live literature (relying on static knowledge graphs or hallucinated citations), (ii) experimental numbers are fabricated rather than executed (CycleResearcher explicitly acknowledges synthetic results), and (iii) there is no standardized multi-dimensional quality framework to assess whether AI-generated manuscripts meet publication standards. The paper targets all three gaps simultaneously.

Method¶

RLEv4 is a five-stage pipeline with a leader-worker LLM routing scheme (deepseek-v4-pro for reasoning/judging, deepseek-chat for structured writing).

Stage 1 — Literature acquisition. A section-aware relevance score $$R(p,t) = \sum_{s \in S} w_s \cdot \cos\!\bigl(e_s(p),\, e_q(t)\bigr)$$ is computed via SPECTER2 embeddings (BM25+ fallback), with section weights: abstract 0.30, methods 0.25, results 0.20, introduction 0.12, discussion 0.08, conclusion 0.05. Two snowball iterations via the Semantic Scholar batch API expand an initial 60-paper seed to 60–100 papers above threshold $\tau \in [0.05, 0.08]$.

Stage 2 — Knowledge graph and claim alignment. A directed citation graph $G=(V,E)$ is built; PageRank identifies influential papers. Up to five falsifiable claims per paper are extracted and classified pairwise as SUPPORTS / CONTRADICTS / ORTHOGONAL, stored in SQLite, and used to drive the manuscript's gap analysis.

Stage 3 — Manuscript generation. The leader produces a JSON architecture plan; the worker drafts sections sequentially under structured prompts enforcing active voice and precise numerical reporting.

Stage 4 — Autonomous bioinformatics experimentation. A coding agent writes, executes, and debugs real Python scripts (up to 12 attempts, 1-hour timeout) performing progressive multiple-sequence alignment with Jukes-Cantor distances $d = -\tfrac{3}{4}\ln\!\bigl(1-\tfrac{4p}{3}\bigr)$, and statistical validation via an injected rigor.py module (5,000-shuffle permutation tests, bootstrap CIs, Benjamini-Hochberg correction, Cohen's $d$, null-baseline comparisons). All outputs are saved to results.json and injected verbatim into every manuscript section.

Stage 5 — Three-tier hybrid quality scorer + improvement loop. Eight dimensions (novelty, contribution, soundness, presentation, reproducibility, grounding, gap relevance, structural completeness) are scored by: - Tier 1 (55%): G-Eval LLM-as-judge with z-normalisation against approximate ICLR 2023–2025 reference calibrations: $T_d = \text{clip}\!\bigl(50 + 16.7\,\tfrac{r_d - \mu_d}{\sigma_d},\,0,\,100\bigr)$ - Tier 2 (25%): corpus entity-network (anchor coverage $c$, reference diversity $\rho$): $b = 100(0.6c + 0.4\rho)$ - Tier 3 (20%): heuristic surface features (word count, table count, math density)

A hallucination penalty $P_h = \min\!\bigl(1,\,0.5\,b_\text{cite} + 0.3\min(1,0.05\,n_\text{mis}) + 0.2\max(0, 0.30-\kappa)\bigr)$ is applied to grounding and soundness, and the overall score becomes $Q = \tfrac{1}{8}\sum_d s_d \cdot (1 - 0.3 P_h)$.

The improvement loop runs for 60 iterations with three routing actions (ADD_ANALYSIS, GATHER_EVIDENCE, REWRITE) selected per the weakest dimension, with escalation on failure streaks and compacted revision memory. Every $K = \lfloor\text{IMPROVE\_ITERATIONS}/6\rfloor$ iterations (default $K=10$) a deep research cycle fires: gap finding → new experiment execution → full re-manuscripting, adopted only if the score does not regress by more than 0.01. Score stabilisation re-judges only dimensions affected by each revision to prevent judge variance from masking real gains.

Key Contributions¶

Section-aware SPECTER2-based relevance scoring with two-round snowball citation expansion, yielding zero out-of-range citations across all five runs
Autonomous coding agent executing real computational biology experiments (not synthetic), with rigor.py-injected statistical validation and a pre-execution AST syntax gate
Three-tier hybrid quality scorer combining LLM-as-judge, corpus entity-network, and heuristic features, with a deterministic hallucination audit penalty
Researcher-action routing (ADD_ANALYSIS / GATHER_EVIDENCE / REWRITE) with failure-streak escalation and compacted revision memory preventing doom-loops
Deep research cycles that expand scientific content by re-running experiments on identified gaps, not merely polishing prose
Score stabilisation mechanism that re-judges only changed dimensions, preventing judge variance from overriding genuine improvements
LaTeX export with eight-stage escalating compile pipeline guaranteeing a PDF is always produced

Results¶

Average quality improvement: +17.96 points (0–100 scale) across five bioinformatics case studies; maximum +26.04 (CpG Island Detection)
Initial score range: 37.16–51.84; final score range: 60.88–68.55 (B to B+ band), suggesting the loop normalises across problem types
All five runs produced compiled PDFs with zero out-of-range citations
Three independent LLMs (cross-model replication) converged on the same quality ranking
Human reviewer average: 7.0/10 across five manuscripts
Cost: approximately $0.31/paper; runtime: 100–190 minutes at 60 iterations with deep cycles
Strongest final dimensions (averaged): gap relevance 77.71, grounding 75.01, reproducibility 74.28, contribution 71.50

Limitations¶

Evaluated on only five bioinformatics case studies; no held-out test set or comparison against a human-written baseline from the same prompts
Final scores of 60–68/100 (B/B+ grade) fall short of the A-range (≥85) indicating the output is not yet at publication quality
External human validation is a single reviewer scoring five manuscripts — statistically weak
ICLR reference calibrations ($\mu_d, \sigma_d$) are described as "approximate," making z-normalisation potentially miscalibrated
Deep research cycles fire at fixed intervals ($K=10$) rather than adaptively in response to quality plateaus
Hardcoded sequences and matrices for experiments; real NCBI Entrez fallbacks are used only with retry wrappers, limiting generality beyond pre-specified bioinformatics problems
System is domain-locked to computational biology; generalization to other sciences is undemonstrated
No adversarial evaluation of the hallucination penalty (i.e., whether $P_h$ correctly detects all fabricated claims)

Relevance to Harnesses / Meta-Harnesses¶

RLEv4 is a textbook multi-agent meta-harness: it orchestrates a chain of specialized sub-agents (literature retriever, knowledge-graph builder, planning agent, autonomous coding agent, LLM-as-judge scorer, context-rich reviser) under a deterministic control loop with explicit routing logic. The improvement loop's action-routing mechanism — dispatching to ADD_ANALYSIS, GATHER_EVIDENCE, or REWRITE based on quality signals, with escalation ladders and memory compaction — is a reusable pattern for any iterative multi-agent refinement harness beyond manuscript generation. The leader-worker model-tier split (reasoning model for judging/synthesis, fast chat model for structured output) and the score-stabilisation trick (re-judging only affected dimensions) are both directly applicable engineering lessons for meta-harnesses that use LLM-as-judge feedback loops. The deep research cycle — a scheduled sub-workflow that re-invokes the experiment and retrieval agents to expand content rather than merely rewrite it — illustrates how a meta-harness can escape local optima by periodically resetting its context with new grounded evidence.