AtomiMed: Hierarchical Atomic Fact-Checking for Universal Clinical-Aware Medical Report Evaluation¶

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AtomiMed is a modality-agnostic evaluation framework for Medical Report Generation (MRG) that decomposes clinical narratives into a two-level hierarchy of Atomic Clinical Facts (ACFs) and verifies them via a bidirectional Agentic Cross-Verification loop. It outperforms all prior metrics—including LLM-as-a-Judge GREEN—in correlation with radiologist judgment across X-ray, CT, MRI, and Ultrasound. The accompanying OmniMRG-Bench provides 178K+ expert-verified QA pairs spanning four imaging modalities.

Problem¶

Existing MRG evaluation metrics fail along multiple axes simultaneously: lexical metrics (BLEU, ROUGE) are semantically blind and treat "no pleural effusion" and "pleural effusion" as near-identical; specialized graph-based metrics (CheXbert, RadGraph) are locked to chest X-ray and cannot generalize across modalities; LLM-as-a-Judge approaches (GREEN) lack per-finding audit traces and degrade severely outside their chest-centric training distribution. No prior metric jointly achieves modality universality, attribute-level granularity, and interpretable error attribution.

Method¶

AtomiMed operates in two stages:

Stage 1 — Hierarchical Atomic Decomposition. Given a report $R$, an instruction-tuned LLM (Qwen3-235B-A22B, $T=0$) extracts: - Disease-level QA: binary presence/absence pairs $\{(q_i, a_i)\}_{i=1}^N$, $a_i \in \{\text{yes}, \text{no}\}$ - Attribute-level QA: per-finding descriptors (location, size, morphology, severity, quantity, temporal change) formalized as $\{(d_k, \{(q_{k,j}, a_{k,j})\}_{j=1}^{M_k})\}_{k=1}^K$

Output is constrained to valid JSON via a fixed prompt template.

Stage 2 — Agentic Cross-Verification. A bidirectional loop uses the same LLM as an evidence reader: in the GT→INF direction it measures recall (did the generated report contain the ground-truth facts?); in the INF→GT direction it measures precision. Scores: $$P_{\text{dis}} = \frac{C_{\text{inf}\to\text{gt}}}{N_{\text{inf}}}, \quad R_{\text{dis}} = \frac{C_{\text{gt}\to\text{inf}}}{N_{\text{gt}}}, \quad F1_{\text{dis}} = \frac{2P_{\text{dis}}R_{\text{dis}}}{P_{\text{dis}}+R_{\text{dis}}}$$ Attribute scoring is conditioned on disease-level alignment (fuzzy string matching at threshold $\theta=0.8$). Final score equally averages both levels: $$F1 = \frac{2PR}{P+R}, \quad P = \tfrac{1}{2}P_{\text{dis}} + \tfrac{1}{2}P_{\text{attr}}, \quad R = \tfrac{1}{2}R_{\text{dis}} + \tfrac{1}{2}R_{\text{attr}}$$

Key Contributions¶

AtomiMed: bidirectional agentic cross-verification framework decomposing reports into Disease- and Attribute-level ACFs, enabling decoupled scoring of diagnostic detection vs. descriptive accuracy with per-finding audit traces.
OmniMRG-Bench: first multi-modal MRG benchmark across X-ray, CT, MRI, and Ultrasound; 9 anatomical systems, 6 attribute categories, 178K+ expert-verified hierarchical QA pairs.
MRGEvalKit: open-source toolkit for reproducible hierarchical atomic scoring.
Empirical demonstration that holistic metrics (including GREEN) exhibit severe cross-modal degradation invisible to prior evaluations.

Results¶

ReXVal (X-ray, error-count correlation): AtomiMed Spearman $\rho = 0.806$, Kendall $\tau = 0.642$, vs. GREEN $\rho = 0.798$, $\tau = 0.626$.
ReFiSco-v0: $\rho = 0.744$ vs. GREEN $\rho = 0.709$.
RadEvalX: $\rho = 0.561$ vs. GREEN $\rho = 0.539$.
Pairwise preference (X-ray): AtomiMed ACC = 95.71%, $\tau = 0.9807$, MAE = 0.0214 vs. GREEN ACC = 63.57%, $\tau = 0.6481$, MAE = 0.1857.
Pairwise preference (CT): ACC = 84.33%, $\tau = 0.7403$ vs. GREEN ACC = 47.14%, $\tau = 0.3283$.
Pairwise preference (MRI): $\tau = 0.4500$ vs. GREEN $\tau = 0.1513$ (near-random).
Pairwise preference (Ultrasound): ACC = 49.86% vs. GREEN 33.83%, RaTEScore 35.00%.
Attribute heatmaps reveal systematic model failures: Severity scores near floor (HuatuoGPT-34B: 1.27, Qwen2.5VL-7B: 0.80); Digestive, Reproductive, and Urinary systems severely underserved across all tested models.

Limitations¶

Inference cost is substantial: backbone is Qwen3-235B-A22B run at temperature 0 for every decomposition and every cross-verification query; the paper acknowledges this and defers distillation to future work.
RaTE-Eval shows AtomiMed ($\rho = 0.413$) underperforming GREEN ($\rho = 0.457$), suggesting the atomic decomposition approach does not universally dominate.
Attribute matching relies on heuristic fuzzy string similarity ($\theta = 0.8$), which may introduce false alignments or misses for synonymous clinical terminology.
Benchmark coverage is still limited: no longitudinal imaging, no pathology reports, no clinical specialties beyond radiology.
The MRGEvalKit extraction quality depends on the same LLM backbone used for scoring, creating potential circularity if the backbone has systematic blind spots.

Relevance to Agentic AI / LLM Agents¶

AtomiMed is a concrete application of the LLM-as-agent evaluator paradigm: rather than using an LLM as a monolithic judge, it deploys an LLM in a structured agentic loop—decompose, query, verify bidirectionally—that operationalizes a multi-step peer-review workflow. This is directly relevant to work on agentic fact-checking, tool-augmented verification, and structured LLM reasoning pipelines, where decomposing complex tasks into atomic subtasks and iterating over them improves both reliability and interpretability. The bidirectional cross-verification design also exemplifies a broader pattern in multi-agent systems: using the same model in complementary roles (precision vs. recall oracle) to produce richer signal than a single forward pass. For researchers building agentic systems that must self-evaluate or audit their outputs in high-stakes domains, AtomiMed's architecture offers a transferable blueprint.