Skip to content

CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; CARE: evidence-grounded agentic framework for clinical medical reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CARE is an agentic framework for multimodal medical VQA that decomposes monolithic VLM inference into three specialist modules — entity proposal, referring segmentation, and evidence-grounded VQA — coordinated by an LLM planner. By feeding pixel-level ROI evidence back into reasoning and using RLVR to align each module, CARE-Flow (10B) outperforms same-size SOTA by 10.9% and CARE-Coord outperforms the heavily-trained 32B Lingshu baseline by 5.2%.

Problem

Existing medical VLMs operate as end-to-end black boxes that map image+text directly to answers without explicitly localizing supporting visual findings. This design invites shortcut learning and hallucination because no grounded evidence is retrieved or verified. Approaches that add grounding typically treat it as an isolated perception head whose outputs are not fed back into reasoning, and coupling grounding with reasoning inside a single generalist model amplifies error propagation while requiring large-scale fine-grained annotated data and expensive multi-turn RL.

Method

CARE decomposes medical VQA into three coordinated specialist modules:

  1. Medical Entity Proposal — A compact InternVL3-2B VLM is prompted with the user's question to propose relevant anatomical entities. It is fine-tuned with RLVR (DAPO algorithm) using a composite reward: an embedding-similarity reward \(R_{\text{sim}}\) computed via the Kuhn–Munkres algorithm over cosine similarities between predicted and ground-truth entity embeddings (Equation 1), plus a count reward \(R_{\text{count}}\) and repetition penalty \(R_{\text{repetition}}\).

  2. Entity Referring Segmentation — A SA-Med-2D-based model fuses a frozen biomedical BERT text encoder with SAM layers via modality-token-conditioned concatenation. Given an entity name, it produces a pixel-level mask \(M = \text{Dec}_M(\text{Enc}_{\text{SAM}}(t)[0:|t_I|],\, \text{Proj}_T(t_T))\) and a confidence score \(C = 1 - \text{Entropy}(M_p)/\log(2)\); masks below threshold \(\tau_C = 70\%\) are discarded.

  3. Evidence-Grounded VQA (EG-VQA) — An InternVL3-2B/8B VLM receives the original image augmented with one of three evidence views: (i) zoom-in crop, (ii) binary mask, or (iii) global all-ones mask. It is trained with SFT + DAPO using a reward \(R_{\text{EG-VQA}} = R_{\text{acc}} + R_{\text{format}} + R_{\text{length}}\).

Two operating modes are provided: CARE-Flow executes all three evidence views and uses majority vote; CARE-Coord uses a proprietary LLM coordinator (GPT-5) that plans tool invocations, selects the optimal evidence view, and performs iterative CoT-answer consistency review post-inference.

Key Contributions

  • First medical agentic framework with a dynamic coordinator that plans tool use and iteratively reviews CoT-answer consistency for clinical accountability.
  • Decoupled specialist architecture that feeds pixel-level ROI evidence (segmentation masks, zoom-in crops, global indicators) back into VQA reasoning, reducing shortcut learning without requiring paired ROI-VQA supervision.
  • RLVR fine-tuning (DAPO) for each specialist module with task-specific verifiable rewards, including a soft embedding-similarity reward for entity proposal that handles domain shift from synthetic training data.
  • Confidence-gated segmentation pass-through: low-confidence masks (\(C < \tau_C\)) fall back to global evidence, preventing bad segmentations from corrupting downstream reasoning.
  • Synthetic data generation pipeline for entity proposal training (10k pairs from SA-Med-20M) to bootstrap a task with no public dataset.

Results

  • CARE-Flow-B (10B): 74.91% average accuracy across four benchmarks, vs. 64.07% for GPT-4o and 66.66% for Lingshu-7B — a +10.9% gain over the same-size (10B) SOTA.
  • CARE-Coord-B: 77.54% average, outperforming Lingshu-32B (72.29%) by +5.2% despite using far fewer parameters in its specialist VLMs.
  • CARE-Flow-S (4B): 70.72% average — competitive with 32B general VLMs (InternVL3-38B: 67.26%).
  • Per-benchmark highlights for CARE-Coord-B: OMVQA-3k 97.97%, VQA-RAD 68.29%, SLAKE 83.11%, VQA-Med-2019 60.80%.
  • Ablation (Table 2): adding all three evidence types during training yields +2.5% over mask-only; adding coordinator planning+review yields a further +2.6%, totaling +5.1% over the baseline row.
  • Ablation (Table 3): SFT+DAPO+\(R_{\text{length}}\) (CARE-Flow) outperforms SFT alone by +9.6% overall.
  • Coordinator ablation (Table 4): GPT-5 coordinator (77.5%) outperforms InternVL3-38B (74.0%) and GPT-4o (73.3%); a fine-tuned InternVL3-8B coordinator achieves 75.1%, approaching GPT-4o.

Limitations

  • The CARE-Coord default coordinator is GPT-5 (proprietary), making the full system dependent on an external API with associated cost and latency.
  • Segmentation model training relies on synthetic entity-question pairs derived from SA-Med-20M; domain gap between synthetic and real clinical queries may remain.
  • Out-of-domain (OOD) gains on VQA-Med-2019 are smaller than in-domain gains (60.80% vs. 97.97% on OMVQA), suggesting the framework still benefits from in-domain distribution.
  • Iterative CoT-answer review by the coordinator adds inference-time overhead proportional to the number of inconsistent thought-answer pairs.
  • Segmentation confidence thresholding (\(\tau_C = 70\%\)) is empirically calibrated; failure modes for entities where the segmentation model systematically underestimates confidence are not analyzed.

Relevance to Agentic AI / LLM Agents

CARE is a concrete instantiation of the specialist-coordinator agentic pattern: a planner LLM dynamically invokes domain-expert tools, selects evidence views, and performs post-hoc verification — a paradigm directly generalizable beyond medicine to any domain requiring grounded, multi-step reasoning. The iterative CoT-answer consistency review is a lightweight alternative to full LLM-verifier loops, relevant to anyone designing cost-efficient agentic pipelines. The use of RLVR (DAPO) to align each specialist module independently, rather than end-to-end, demonstrates how to train modular agents without requiring joint supervision across the full pipeline. The confidence-gated tool-call pass-through is a practical design pattern for graceful degradation in agentic tool use when sub-agent outputs are unreliable.