Skip to content

CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework

🕒 Published (v1): 2026-03-02 08:38 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CARE is an evidence-grounded agentic framework for multi-modal medical VQA that decomposes end-to-end black-box VLM inference into three coordinated specialist stages: entity proposal, pixel-level referring segmentation, and grounded VQA. A VLM coordinator plans tool invocations and performs iterative chain-of-thought/answer consistency review. CARE-Flow (10B) improves average accuracy by 10.9% over same-size SOTA; adding the coordinator (CARE-Coord) gains a further 2.6% for a 5.2% margin over the 32B heavily-trained baseline.

Problem

Monolithic, single-shot VLMs for medical image VQA skip explicit visual evidence retrieval, causing shortcut learning and hallucination, especially under distribution shift. Existing grounded VLMs either treat grounding as an auxiliary output head without feeding it back into reasoning, or couple all steps inside one generalist model—amplifying error propagation and requiring expensive paired ROI-VQA supervision that is scarce in medicine.

Method

CARE decomposes inference into three specialist sub-tasks and coordinates them either statically (CARE-Flow) or dynamically (CARE-Coord):

  1. Medical Entity Proposal: A compact InternVL3-2B VLM, fine-tuned with RLVR (DAPO algorithm), proposes candidate anatomical structures conditioned on the user query. The reward combines a soft embedding-similarity score \(R_{\text{sim}}\) via Kuhn–Munkres optimal matching on cosine similarities, a count reward \(R_{\text{count}}\), a repetition penalty \(R_{\text{repetition}}\), and a format reward \(R_{\text{format}}\).

  2. Entity Referring Segmentation: A SA-Med-2D-based model (~600M) augmented with a frozen BioBERT text encoder and a learned text projector. For entity \(\hat{e}\) and image \(x_I\), the segmentation mask is \(M = \text{Dec}_M\!\bigl(\text{Enc}_\text{SAM}(\text{concat}(t_I, t_T) + t_\text{mod})[0{:}|t_I|],\ \text{Proj}_T(t_T)\bigr)\). A mask-entropy confidence score \(C = 1 - \frac{\text{Entropy}(M_p)}{\log 2}\) gates downstream use; masks with \(C < \tau_C = 70\%\) are dropped.

  3. Evidence-Grounded VQA (EG-VQA): An InternVL3-2B/8B VLM fine-tuned (SFT then DAPO RFT) to reason over the full image plus one of three evidence views—zoom-in crop, binary mask, or global all-ones mask. A CoT-length reward \(R_\text{length} = 0.25 \cdot \min(1, |\hat{y}|/L)\) is added alongside binary accuracy and format rewards.

CARE-Flow runs the pipeline statically with majority vote over all three evidence types. CARE-Coord uses a powerful VLM (GPT-5 in default config) that plans tool calls, selects the optimal evidence view, and iteratively checks CoT–answer consistency, re-running expert models when mismatches are detected.

Key Contributions

  • First medical agentic framework for accountable VQA that feeds pixel-level referring-segmentation evidence back into a downstream reasoning VLM.
  • Soft embedding-similarity reward (with Hungarian matching) for RLVR-based entity proposal, avoiding zero-gradient issues from exact-match rewards on synthetic training data.
  • Three-evidence-view EG-VQA design (zoom-in / binary mask / global indicator) with entropy-based confidence gating to prevent low-quality segmentation from degrading answers.
  • CARE-Coord: a dynamic coordinator that plans tool invocations, selects evidence views, and performs iterative CoT–answer consistency review without an external LLM verifier at every step.
  • CARE-Flow (10B) surpasses InternVL3-8B by >9 points average and Lingshu-32B by 2.26%; CARE-Coord achieves 77.54% average across four benchmarks, +5.2% over Lingshu-32B.

Results

  • CARE-Flow-B (10B) vs. InternVL3-8B (same-tier baseline): 74.91% vs. 65.34% overall (+9.57 pp).
  • CARE-Flow-B vs. Lingshu-32B (heavily-trained SOTA): 74.91% vs. 72.29% (+2.62 pp).
  • CARE-Coord-B vs. Lingshu-32B: 77.54% vs. 72.29% (+5.25 pp).
  • Per-benchmark highlights (CARE-Coord-B): OMVQA-3k 97.97%, VQA-RAD 68.29%, SLAKE 83.11%, VQA-Med-2019 60.80%.
  • CARE-Flow-S (4B) overall: 70.72%, surpassing all general-purpose open-source models ≤32B.
  • GPT-5 as coordinator vs. no coordinator (ablation Table 4): 77.5% vs. 74.9% overall (+2.6 pp).
  • SFT + DAPO + \(R_\text{length}\) vs. SFT only (ablation Table 3): 74.9% vs. 65.3% overall (+9.6 pp).
  • Adding coordinator review over planning alone (ablation Table 2): +2.6 pp overall (row 6 → row 7).

Limitations

  • The default CARE-Coord relies on GPT-5, a proprietary model, introducing cost and external API dependency; training a small open-source coordinator (InternVL3-8B + RFT) reaches only 75.1% vs. 77.5%.
  • Entity proposal and segmentation training data are synthesized from SA-Med-20M; domain shift from synthetic proposals to real user questions is partially mitigated by soft rewards but not fully characterized.
  • Inference latency increases proportionally with tool calls; the coordinator can skip steps for global questions but multi-turn review still adds overhead not quantified in the paper.
  • Out-of-domain (OOD) performance (VQA-Med-2019) lags significantly behind in-domain scores (e.g., 60.80% OOD vs. 97.97% OMVQA), indicating limited cross-dataset generalization.
  • Evaluation of open-ended questions relies on GPT-4o as the judge, introducing a secondary model dependency and potential scoring variance.

Relevance to Harnesses / Meta-Harnesses

CARE is a concrete medical-domain instance of a multi-stage specialist harness with a VLM meta-coordinator: it decouples perception (entity proposal + segmentation) from reasoning (EG-VQA), chains them with typed inter-module contracts (entity names → masks + confidence scores → evidence-augmented prompts), and adds a coordinator layer that dynamically selects tools, routes evidence views, and iteratively verifies sub-module outputs—all hallmarks of a meta-harness architecture. The CARE-Flow / CARE-Coord duality directly illustrates the design tradeoff between static pipelines and dynamic agentic orchestration discussed in meta-harness literature. The iterative CoT–answer consistency review is a lightweight self-verification loop that substitutes for a full LLM verifier, an engineering pattern applicable to any multi-module harness. Reinforcement-learning-with-verifiable-rewards applied per specialist module (rather than end-to-end) offers a training recipe for harnesses where each sub-agent must be independently optimizable.