Skip to content

DeepEvidence: Empowering Biomedical Discovery with Deep Knowledge Graph Research

🕒 Published (v1): 2025-12-23 14:34 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeepEvidence is an orchestrator-plus-subagent AI framework that performs "Deep Research" over heterogeneous biomedical knowledge graphs (KGs) by coordinating two complementary exploration agents—Breadth-First ReSearch (BFRS) and Depth-First ReSearch (DFRS)—while maintaining an incrementally built evidence graph as shared memory. It substantially outperforms frontier LLMs and general-purpose biomedical agents on four established benchmarks and six curated lifecycle tasks spanning drug discovery through evidence-based medicine.

Problem

Biomedical knowledge graphs (PubTator, KEGG, OpenTargets, BioThings, ClinicalTrials.gov, etc.) are structurally heterogeneous, continuously evolving, and poorly interoperable. Existing agents rely on internet-scale text and perform shallow, uncoordinated API lookups, leaving multi-hop cross-graph reasoning largely manual. No prior system provided a principled strategy for deep, systematic traversal across heterogeneous KGs to support the full biomedical discovery lifecycle.

Method

DeepEvidence uses a three-layer architecture:

  1. Orchestrator agent: Receives the user query, plans the research sequence (which sub-agents to invoke and in what order), and incrementally maintains an evidence graph—a structured graph of entities (papers, genes, drugs, pathways, trials), their relations (targets, cites, tests), and supporting provenance.

  2. Research sub-agents:

  3. BFRS agent: Issues parallel queries across multiple KG APIs (KEGG, ChEMBL, OpenFDA, OpenTargets, MyGene, etc.) to enumerate first-hop neighbors and candidate entities at breadth.

  4. DFRS agent: Takes high-priority entities from BFRS and performs multi-hop traversal (e.g., tracing citation networks recursively) for evidence-intensive tasks. Agents autonomously select which KGs to query and what code to execute; they can bridge across graphs by recognizing shared entities ("bridge entities").

  5. Execution sandbox: Agents write and execute Python code within a sandboxed environment to query APIs at scale, persist results as structured files (spreadsheets), and perform downstream analysis—enabling reproducible, multi-step programmatic research workflows.

The orchestrator dynamically adjusts BFRS/DFRS interleaving based on the query; users can steer by specifying relevant domains or expert priors.

Key Contributions

  • Orchestrator–subagent harness design with BFRS/DFRS dual-mode KG traversal, purpose-built for heterogeneous biomedical graphs.
  • Incrementally constructed evidence graph serving as persistent, inspectable agent memory across multi-step research workflows.
  • Unified API interface layer spanning publications, genes, chemicals, drugs, diseases, phenotypes, proteins, and clinical trials with an execution sandbox for programmatic retrieval and analysis.
  • Six new benchmark tasks covering the full drug development lifecycle (drug discovery, preclinical, clinical trial design, evidence-based medicine).
  • Demonstrated cross-graph "bridge entity" recognition enabling cross-resource multi-hop reasoning without prior alignment.

Results

  • HLE-Medicine: DeepEvidence 40.0% vs. Biomni 20.0%, ToolUniverse 10.0%, Sonnet-4.5 3.3%, GPT-5 3.3%.
  • LabBench-LitQA2: 80.0% vs. Biomni 32.0%, Sonnet-4.5 48.0%.
  • SuperGPQA-Medicine-Hard: 47.1% vs. Biomni 40.7%, Sonnet-4.5 43.6%.
  • TrialPanorama-EvidenceQA: 96.0% vs. Biomni 84.0%, Sonnet-4.5 88.0%.
  • Target identification: 68% vs. Biomni 56%, ToolUniverse 40%, LLMs 42%.
  • MOA/pathway reasoning: 72% vs. Biomni 44%, ToolUniverse 44%, LLMs 52%.
  • In vivo metabolic flux response: 80% vs. Biomni 68%, LLMs 60%, ToolUniverse 52%.
  • Sample size estimation: 68% vs. Biomni 20%, ToolUniverse 32%, PubMed-search LLM 24%.
  • Drug regimen design: 52% vs. Biomni 36%, ToolUniverse 28%, PubMed-search LLM 20%.
  • Surrogate endpoint discovery (F1): 73.3% vs. Biomni 60.9%, ToolUniverse 61.9%, LLMs 58.7%.
  • Evidence gap detection rate@30: 90.0% vs. Biomni 50.0%, ToolUniverse 15.0%, LLMs 10.0%.
  • Evidence gap recall@30: 44.14% vs. Biomni 30.17%, ToolUniverse 5.06%, LLMs 2.67%.
  • All baselines run with the same underlying LLM (GPT-5).

Limitations

  • KG coverage is incomplete; benchmarks represent only a subset of biomedical research tasks.
  • Requires well-curated KGs with stable schemas and reliable APIs (KEGG, Gene Ontology); cannot yet handle proprietary, internally hosted, or schema-inconsistent graphs.
  • BFRS/DFRS traversal strategies are fixed heuristics rather than learned policies—unlikely to be optimal across all task types.
  • Has not reached expert-level performance on many tasks; relies on the strength of the underlying LLM.
  • Text-only modalities; multimodal data (imaging, structural biology, proteomics) not yet supported.

Relevance to Harnesses / Meta-Harnesses

DeepEvidence is a canonical domain-specialized multi-agent harness: an orchestrator dynamically dispatches typed sub-agents (BFRS, DFRS), each with their own tool set and execution sandbox, coordinated through shared persistent state (the evidence graph). The architecture cleanly instantiates the orchestrator-subagent pattern discussed in meta-harness literature—with the added twist that the shared memory is itself a structured knowledge artifact rather than a flat context window. The execution sandbox, where agents generate and run code to query APIs and persist structured files, is directly analogous to tool-use harnesses (e.g., code-interpreter loops) embedded within a larger planning loop. For practitioners building harnesses over heterogeneous data sources, DeepEvidence demonstrates how to decompose tool selection and traversal strategy into complementary agent roles and how to use an incrementally constructed graph as a lightweight, inspectable memory layer shared across subagents.