MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents¶

🕒 Published (v1): 2026-07-03 02:24 UTC · Source: Arxiv · link

Why this paper was selected

Medical calculator agents with implicit tool selection; practical benchmark for clinical agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MedCalc-Pro introduces a benchmark of 2,268 real-world clinical cases across three task tiers (single-calculator, multi-calculator, nested-calculator) and an accompanying four-stage agent harness that orchestrates multi-tool selection, dependency-aware nested-tool invocation, and structured parameter validation for medical scoring. Existing benchmarks and frameworks collapse on fuzzy, goal-driven queries requiring multiple interdependent calculators; MedCalc-Pro exposes this failure and provides a systematic evaluation scaffold.

Problem¶

Current medical calculation benchmarks (MedCalc-Bench, MeNTi, MedRaC, CalcQA) assume single-calculator, explicit-query settings where the target tool is named in the prompt. Real clinical workflows require: (1) selecting multiple calculators from goal-oriented queries that never name the tool, and (2) resolving nested dependencies where a parent calculator (e.g., APACHE II) requires the output of a sub-calculator (e.g., GCS) as an input. No existing benchmark covers all three tiers, and no existing agent framework handles nested-tool calling with error propagation suppression.

Method¶

The framework is a sequential four-stage pipeline:

Query Rewriting: Rewrites the original fuzzy clinical query into four parallel representations—original, clinical context, key parameters, and core functionality—using an LLM, broadening the retrieval surface.
Retrieval and Reranking: Each of the four query variants independently retrieves top-$K_{\text{route}}=16$ candidate calculators via dense embedding (m3e-base, cosine similarity). Candidates are merged using Reciprocal Rank Fusion: $$\text{RRF}(d) = \sum_{i=1}^{4} \frac{1}{k + r_i(d)}$$ yielding a final candidate pool of $K_{\text{cand}}=32$ tools.
Tool Selection: The LLM receives full descriptions and parameter schemas for all $K_{\text{cand}}$ candidates and freely selects the applicable subset (no forced top-1).
Tool Execution: Identifies inter-calculator dependencies; executes sub-calculators first and feeds their outputs into parent calculators. After parameter extraction, a four-check structured validation pipeline runs: Completeness → Type → Enum → Unit. Failures trigger backtracking to parameter extraction. An evidence audit then cross-checks extracted values against the original clinical text before final computation.

For nested-calculator tasks the benchmark covers 77 tools across 14 clinical departments; quality control requires 95%+ annotation accuracy via three-stage clinician review.

Key Contributions¶

MedCalc-Pro benchmark: 2,268 cases, 77 calculators, 14 departments; first benchmark with multi-calculator and nested-calculator task tiers and fuzzy goal-driven queries.
Multi-dimensional query rewriting: LLM-generated parallel query variants (clinical context, key parameters, core functionality) + RRF reranking to recover correct tools from implicit clinical intent.
Dependency-aware nested-tool execution: Automatic detection of parent–sub-calculator chains and topological execution ordering.
Structured parameter validation + evidence audit: Four-check pipeline with backtracking to suppress error propagation in nested chains.
Systematic model comparison: Six LLMs × four frameworks (MeNTi, MedRaC, ReAct, Ours) across all three task tiers.

Results¶

Single-calculator: Framework achieves R-F1 99.62%, Score Accuracy 80.15% (gpt-oss-120b), vs. best baseline MedRaC at 86.57% R-F1 and 76.72% Score Accuracy (+13.05% R-F1, +4.77% Score).
Multi-calculator: Framework achieves 59.97% R-F1, 59.53% Score Accuracy vs. best baseline ReAct at 54.32% R-F1, 37.57% Score Accuracy (+19.07% Extract, +21.96% Score).
Nested-calculator: Framework improves Sub Overall by +32.27% and Parent Overall by +14.51% over best baseline MeNTi.
MedRaC failure: MedRaC scores 0.00% on all nested-calculator metrics due to RAG retrieval failing under fuzzy queries without explicit calculator names.
Ablation: Removing retrieval/reranking causes the largest single-calculator drop (−19.96% R-F1); removing structured validation causes the largest nested-calculator drop (−19.70% Extract, −11.3% Parent Overall).
Hyperparameter: $K_{\text{cand}}=32$ is the cost-optimal setting; increasing to 64 yields only +1.16% F1 at ~2× token cost.

Limitations¶

Nested-calculator gains at the sub-calculator level only partially propagate to parent-level outputs; residual intermediate errors still degrade final scores.
Benchmark size for nested tasks is small (62 cases), limiting statistical power.
Query rewriting and the tool selection stage add significant token overhead (prompt length scales with $K_{\text{cand}}$).
Evaluation is conducted on a fixed embedding model (m3e-base); retrieval quality may degrade on out-of-distribution calculator descriptions.
The framework is evaluated only on medical calculators; generalizability to other structured tool domains is undemonstrated.

Relevance to Harnesses / Meta-Harnesses¶

MedCalc-Pro's agent framework is a domain-specific harness that implements multi-stage orchestration of heterogeneous tools with explicit dependency resolution—exactly the architectural pattern meta-harnesses are designed to generalize. The four-stage pipeline (rewrite → retrieve → select → execute with validation) demonstrates how a harness can decouple retrieval, planning, and execution while using structured validation gates to prevent error propagation between stages. The nested-calculator execution model (topological sub-tool ordering, output threading) is a concrete implementation of the tool-composition problem that meta-harness designers must solve generically. The comparison against ReAct and MeNTi quantifies the cost of not having a dedicated orchestration harness in multi-step, multi-tool settings.