Skip to content

Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answering

🕒 Published (v1): 2025-11-19 03:08 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GeneGPT's monolithic pipeline over NCBI APIs is irreproducible because it depends on the deprecated Codex model; this paper first reproduces it with open-source LLMs (Llama 3.1, Qwen2.5, Qwen2.5-Coder), documents systematic failure modes, then replaces it with OpenBioLLM—a modular multi-agent framework with role-specialized agents for tool routing, query generation, and response validation. OpenBioLLM matches or outperforms GeneGPT on 11 of 12 benchmark tasks while cutting latency by 40–50%.

Problem

GeneGPT's tool-augmented genomic QA pipeline was built on OpenAI's code-davinci-002, which was deprecated in 2024, rendering the system unreproducible. Its single-agent monolithic design also lacks the structural flexibility needed for multi-hop reasoning (e.g., GeneHop tasks requiring sequential API calls), and its rigid prompt format couples API documentation and in-context demonstrations in ways that cause systematic failures when swapping in open-source models.

Method

Pilot reproduction phase: The GeneGPT prompt-and-pipeline is re-implemented using three open-source backbones (Llama3.1-70B, Qwen2.5-72B, Qwen2.5-Coder-32B) via Ollama + LlamaIndex on A100 GPUs, evaluated on 10-question samples per task to triage model fit. Qwen2.5-72B is selected as the viable backbone; Llama3.1 fails catastrophically (avg 0.19 vs 0.84 for GeneGPT-full).

Prompt engineering: Four targeted fixes applied to Qwen: (1) explicit output-format enforcement (bracketed URLs, Answer: prefix), (2) adding the missing sort=relevance and retmode=json API parameters to prompt documentation, (3) ReAct-style prompting for GeneHop—interleaving Thought/Action/Observation steps with explicit instructions on which JSON fields to extract after each API response.

OpenBioLLM multi-agent framework: Replaces the monolithic inference loop with specialized sub-agents: a tool-routing agent dispatches to the correct NCBI API (E-utils esearch/efetch/esummary, BLAST), a query-generation agent constructs structured API requests, and a response-validation agent performs semantic-level answer extraction and consistency checks. Agents execute with role-specific prompts; LLM-as-evaluator (Qwen2.5-32B, validated against human assessors) is used for tasks requiring semantic scoring (SNP gene function, multi-species alignment, protein-coding genes).

Key Contributions

  • Open-source reproduction of GeneGPT: full pipeline replicated without proprietary models, no fine-tuning.
  • Systematic failure taxonomy (four categories): case-sensitivity in entity matching, relevance-ranking API misuse, query ambiguity, genome assembly coordinate discrepancies.
  • Prompt-level fixes that raise Qwen-72B from 0.89 (10Q pilot) to 0.838 (full 450Q GeneTuring) average accuracy.
  • OpenBioLLM modular multi-agent architecture with role-specialized agents achieving 0.849 GeneTuring / 0.830 GeneHop and 40–50% latency reduction.
  • Empirical finding: smaller role-faithful agents outperform larger models exploiting reasoning shortcuts—role-fit matters more than raw scale.

Results

  • GeneTuring (9 tasks Ă— 50 questions): OpenBioLLM avg 0.849 vs GeneGPT-full 0.833 / GeneGPT-slim 0.804; Qwen-72B with optimized prompt alone: 0.838.
  • GeneHop (3 multi-hop tasks Ă— 50 questions): OpenBioLLM avg 0.830 vs GeneGPT 0.543; Qwen-72B with ReAct prompting alone: 0.729.
  • Matches or outperforms GeneGPT on 11 of 12 tasks.
  • Latency reduced 40–50% across benchmark tasks versus the monolithic GeneGPT pipeline.
  • Qwen-72B with ReAct: SNP gene function 0.93, sequence gene alias 0.71; disease gene location 0.54 (still below GeneGPT's 0.67 for this subtask).
  • LLM-as-evaluator (Qwen2.5-32B) validated against human assessors on GeneGPT responses—agreement confirmed.

Limitations

  • Genome assembly mismatch between NCBI core_nt returns (GRCh37/clone assemblies) and GeneTuring ground truth (hg38) causes systematic coordinate errors regardless of model—an API-level, not model-level, problem.
  • NCBI API retrieves only top 5–10 results; correct answers ranked beyond that window are irrecoverable.
  • Disease gene location task remains below GeneGPT on the 32B variant (0.17 vs 0.67), indicating fragility at smaller agent model sizes.
  • No fine-tuning or domain adaptation; performance ceiling is bounded by what prompt engineering can achieve with general-purpose base models.
  • Paper text is truncated before full OpenBioLLM architecture tables are presented; exact per-agent ablation not fully reported in available text.

Relevance to Harnesses / Meta-Harnesses

OpenBioLLM is a direct example of a domain-specific meta-harness: it wraps a suite of external tool APIs (NCBI E-utils, BLAST) behind a coordinating layer that decomposes tasks and routes them to specialized sub-agents, precisely the orchestration pattern central to harness design. The paper concretizes how role specialization—distinct agents for routing, generation, and validation—resolves the brittleness of a single-agent harness over heterogeneous APIs, a lesson directly transferable to harness architectures in other domains. The ReAct-style Thought/Action/Observation loop embedded in each agent's prompt is a harness-level control mechanism that externalizes reasoning state, enabling traceability and step-level intervention. The 40–50% latency reduction from modularization (vs. a monolithic pipeline) provides empirical evidence for the efficiency case for multi-agent harness decomposition.