Aloe-Vision: Robust Vision-Language Models for Healthcare¶

🕒 Published (v1): 2026-06-25 19:36 UTC · Source: Arxiv · Venue: MIDL 2026 · link

Why this paper was selected

MIDL; robustness-focused healthcare LVLM with scarce-data evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Aloe-Vision is a fully open family of medical large vision-language models (LVLMs) at 7B and 72B scales, trained on a carefully curated 3.57M-sample multimodal mixture (Aloe-Vision-Data) and evaluated against a new contamination-resistant benchmark (CareQA-Vision). The work also introduces an adversarially robust (AR) variant and audits current LVLMs' vulnerability to misleading multimodal inputs, finding significant robustness gaps across all tested models.

Problem¶

Medical LVLM development is bottlenecked by three compounding issues: (1) scarcity of high-quality, diverse medical image-text data; (2) benchmark contamination—widely-used VQA datasets (PathVQA, VQA-RAD, SLAKE) have been public long enough to risk training-evaluation leakage, inflating reported accuracy; and (3) adversarial fragility—current models fail under sycophantic prompts, misleading captions, and incorrect legends, which is unacceptable in clinical deployment. Existing competitive systems (Lingshu, Hulu-Med, GMAI-VL) are only partially or fully closed, blocking reproducible science.

Method¶

Training data (Aloe-Vision-Data): An 8-source SFT mixture of ~3.57M samples / 640M loss tokens, balanced along two axes—modality (multimodal 52%, text-only 48%) and domain (medical 71.5%, general 28.5%). Proportions are computed over loss-contributing tokens (assistant turns only), not raw sample counts, to prevent long-form Chain-of-Thought datasets from dominating gradients. Preprocessing includes: unified conversational schema, Qwen2-VL bounding-box format with coordinates normalized to \([0, 1000)\), sequence-length cap at 4096 tokens, and 64-bit perceptual hashing (pHash) to remove 6,273 near-duplicate evaluation images from training.

Quality filtering: A two-signal semi-automatic filter scores each sample with (a) Qwen2.5-VL-72B-Instruct producing a 1–5 quality score for image/question/answer coherence, and (b) answer perplexity under Qwen2-VL-7B-Instruct—extreme values (very low = trivial, very high = noisy) are flagged. Per-source thresholds are tuned manually; 541,237 samples are removed.

Model training: Single-stage supervised fine-tuning (SFT) of Qwen2-VL-Instruct at 7B and 72B using TRL, optimizing cross-entropy over assistant turns only.

Adversarially robust (AR) variant: A second SFT epoch on 17.2k adversarial samples from a single held-out modality (FracAtlas fractures), testing cross-modal generalization of robustness without touching HEART benchmark modalities.

CareQA-Vision: 301 questions (70 nursing, 231 medicine) derived from Spanish MIR/EIR residency entrance exams (2020–2024), translated to English and converted to open-ended format by Qwen2.5-72B-Instruct with manual verification. Evaluated via majority-vote LLM-as-judge (Qwen2.5-VL-72B, Llama-3.3-70B, Olmo-3-32B); human–judge agreement: Krippendorff's \(\alpha = 0.812\).

Key Contributions¶

Aloe-Vision-Data: First publicly released, ready-to-train 4-way balanced medical LVLM SFT mixture with token-based rebalancing and documented preprocessing.
Aloe-Vision (7B/72B): Fully open medical LVLMs with weights, training recipes, and data; first open medical LVLM at 70B scale.
Aloe-Vision-AR variants: Adversarially robust models trained on a single held-out modality, demonstrating cross-modal robustness transfer.
CareQA-Vision: 301-question contamination-resistant benchmark derived from expert-curated Spanish residency exams.
Systematic adversarial robustness audit of state-of-the-art medical and general LVLMs using the HEART framework (4 attack types: sycophancy, captions, prompt injection, legends).

Results¶

Standard benchmarks (Table 3): - Aloe-Vision-72B-AR achieves the strongest overall large-model performance among open systems, including leads on CareQA-Vision MCQ (75.76), OmniMedVQA (83.97), GMAI-MMBench (55.12), and MMLU (82.52). - Aloe-Vision-72B-AR surpasses Qwen2-VL-72B-Instruct on medical benchmarks (e.g., PathMMU 70.32 vs. 64.71, OMVQA 83.97 vs. 76.09) while matching on general ones (MMLU 82.52 vs. 81.86). - Hulu-Med-7B leads the small-model category (first on 4/10 benchmarks); Aloe-Vision-7B-AR is competitive, ranking first on CareQA-Vision MCQ (60.61) within the small group. - SLAKE results for Lingshu and Hulu-Med are ~15–20% above Aloe-Vision, attributable to training on SLAKE's training split—Aloe-Vision explicitly excludes it. - GPT-5.2 closed-source baseline: CareQA-V MCQ 80.61, PathMMU 68.56, MMLU 86.55—open models remain below on most benchmarks. - Reasoning models (GLM-4.5V, MiMo-VL-7B-RL, Chiron-o1-8B) do not consistently outperform standard fine-tuned models, suggesting current medical VQA tasks are predominantly knowledge-retrieval, not multi-step reasoning.

Adversarial robustness (HEART): - All current LVLMs degrade under adversarial attacks; the AR training stage reduces this degradation. - Cross-modal generalization from FracAtlas-only adversarial fine-tuning is confirmed (details truncated in provided text).

Limitations¶

CareQA-Vision is small (301 questions), limiting statistical power; results on it should be interpreted cautiously.
Adversarial robustness training uses only a single source modality (FracAtlas); generalization across imaging domains is shown but may not be complete.
SLAKE is excluded from training, making direct comparison with Lingshu/Hulu-Med misleading—they trained on SLAKE's training split.
The 72B model is compared only against 32B–34B medical specialists; no 70B-scale medical competitors exist yet.
LLM-as-judge evaluation for open-ended answers carries known biases (e.g., favoring longer answers), acknowledged by the authors.
Dataset coverage (Figure 1) may still have gaps in rare modality-specialty combinations.
General LVLM performance is maintained but not improved, meaning Aloe-Vision does not advance general multimodal capabilities.

Relevance to Vision-Language Models¶

This work directly addresses a reproducibility and reliability deficit in medical VLM research: all training data, weights, and evaluation code are released, providing a public foundation for the community to build on—contrasting with partially-closed state-of-the-art systems like Lingshu and Hulu-Med. The token-based mixture balancing technique and semi-automatic quality filtering pipeline are methodology contributions applicable beyond the medical domain to any domain-specialized VLM. The adversarial robustness findings (all tested LVLMs degrade under sycophancy and caption attacks) have direct implications for deployment safety of VLMs in high-stakes settings. CareQA-Vision addresses the benchmark contamination problem that increasingly undermines VLM progress measurement, offering a template for constructing low-contamination evaluation sets from professional licensure exams.