Skip to content

BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation

🕒 Published (v1): 2025-11-12 01:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BioVerge introduces a benchmark and an LLM agent framework (BioVerge Agent) for biomedical hypothesis generation, combining structured triplet data and textual PubMed literature. The agent uses a ReAct-based loop alternating between Generation and Evaluation modules to iteratively propose and self-assess hypotheses. Key finding: self-evaluation and access to both structured and textual sources measurably improve hypothesis novelty and relevance.

Problem

Biomedical Literature-Based Discovery (LBD) is constrained by single-modality inputs and shallow co-occurrence/graph methods that cannot build multi-hop reasoning chains. LLM agents have been underexplored here due to the absence of standardized benchmarks, curated multi-modal knowledge bases, and execution environments with reproducible evaluation metrics.

Method

BioVerge builds a knowledge base from PubTator3 (10.5M structured hypothesis triplets \((s, r, o)\) with 12 relation types) and PubMed abstracts, all pre-2024 to prevent test leakage. A 177-item diabetes-focused test set drawn from 2024 publications is ranked by journal impact factor.

BioVerge Agent wraps two modules in a ReAct loop (Think → Act → Observe): - Generation module: calls a 9-function API (entity lookup, triplet/relation retrieval, knowledge-graph shortest paths, MeSH hierarchy traversal, article browsing) to propose a hypothesis \((r, d)\). - Evaluation module: scores the proposal on novelty, reasonableness (0–100), and outputs natural-language feedback.

Two architectures are tested: Single Agent (shared memory log) vs. Double Agent (separate memory logs). An Evaluation Threshold (ET \(\in \{30,50,70,90\}\)) gates early termination; at max iterations, an LLM extractor distills the best proposal from the full memory log.

Evaluation metrics: \(\text{novelty}_r\) / \(\text{alignment}_r\) (binary, relation-level) and \(\text{novelty}_d\) / \(\text{alignment}_d\) (LLM-scored 0–100, description-level).

Key Contributions

  • BioVerge benchmark: 10.5M pre-2024 triplets + PubMed literature + searchable knowledge graph; 177-hypothesis test set with temporal cutoff and IF-ranked curation.
  • BioVerge Agent: ReAct-based dual-module (Generate + Evaluate) framework with a structured API enabling tool-augmented multi-modal retrieval.
  • Systematic ablation of agent architecture (Single vs. Double), information source type (relation-only, triplet-only, article-only, KG-only, no evaluation), and ET thresholds.
  • Empirical finding that self-evaluation improves novelty (\(\text{novelty}_r > 98\%\)) and that combining structured + textual sources outperforms either alone.

Results

  • Best configuration: Single Agent, ET=50 → \(\text{alignment}_r = 38.42\%\), \(\text{novelty}_r = 100\%\), \(\text{alignment}_d = 54.10\), \(\text{novelty}_d = 63.39\).
  • Baselines (CoT, Triplet RAG, Article RAG): best relation alignment is CoT at \(36.72\%\); Article RAG achieves highest \(\text{alignment}_d = 57.59\) but lowest \(\text{novelty}_r = 93.22\%\).
  • Single Agent beats all baselines on \(\text{alignment}_r\) (38.42 vs. 36.72 for CoT) while maintaining near-perfect novelty.
  • Double Agent achieves higher \(\text{novelty}_d\) (up to 66.41 at ET=90) but lower \(\text{alignment}_r\) (down to 17.51 at ET=90), reflecting an exploration–accuracy tradeoff.
  • Double Agent uses 8–13 API calls per query vs. 2–4 for Single Agent; outer iteration count grows with ET (4.18→7.87) unlike Single Agent (stable ~1.05–1.26).
  • Ablation: removing the Evaluation module ("Generation Only") drops \(\text{alignment}_r\) to 33.89 and \(\text{alignment}_d\) to 53.93, confirming self-evaluation adds value.

Limitations

  • Test set confined to diabetes mellitus (177 hypotheses), limiting generalizability across disease domains.
  • All experiments use gpt-4o-mini only, due to compute constraints; stronger models may shift the tradeoffs materially.
  • Knowledge cutoff at Jan. 1, 2024 may inadvertently exclude some relevant pre-2024 context due to indexing delays.
  • Single Agent shows causal-relation bias and potential Evaluation module overconfidence (overconfident self-scoring leads to premature termination).
  • Double Agent's exploration diversity comes at high API and compute cost with diminishing alignment returns at high ET.
  • LLM-based \(\text{novelty}_d\) / \(\text{alignment}_d\) evaluators introduce model-dependent scoring variance (high \(\sigma \approx 20\)–34).

Relevance to Harnesses / Meta-Harnesses

BioVerge Agent is a domain-specific agentic harness: it orchestrates a fixed Generate→Evaluate→Refine pipeline over a curated toolset, with the Evaluation Threshold acting as a meta-level termination controller. The Single/Double Agent split is directly a memory-sharing architecture study—how shared vs. isolated state between sub-agents affects exploration diversity and convergence, a core concern for meta-harness designers. The LLM extractor fallback (distilling the best proposal from a full memory log when ET is never reached) is a lightweight meta-evaluation layer analogous to judge-panel patterns in multi-agent harnesses. The systematic ablation over information sources and self-evaluation toggles offers concrete design guidance for harness builders deciding how to scope tool access and when to inject critic agents.