Skip to content

AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering

🕒 Published (v1): 2025-09-26 01:22 UTC · Source: Arxiv · Venue: EMNLP · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AMANDA is a training-free multi-agent framework that wraps any Medical Multimodal LLM (Med-MLLM) with five specialized LLM agents to improve Medical VQA in zero- and few-shot settings. It addresses two reasoning bottlenecks—shallow visual inspection and static pre-trained knowledge—through coarse-to-fine question decomposition and biomedical knowledge graph retrieval. Across eight Med-VQA benchmarks, it consistently improves baseline Med-MLLMs by up to ~19–20% average accuracy without any task-specific fine-tuning.

Problem

Existing Med-MLLMs fail in data-efficient (zero/few-shot) settings due to two bottlenecks: (i) the intrinsic bottleneck—single-step inference misses fine-grained pathological features that require iterative visual examination; (ii) the extrinsic bottleneck—static pre-trained knowledge lacks the specialized, up-to-date domain expertise needed for novel cases, causing hallucinations. Prior agentic approaches (e.g., IdealGPT) are domain-agnostic, while prior med-specific agents require task-specific training.

Method

AMANDA wraps a Med-MLLM with five specialized agents organized into three modules:

Perception Module — A Perceiver agent (the underlying Med-MLLM) generates a medical caption \(c\) and initial answer \(\hat{a}_0\).

Planning Module — A Reasoner synthesizes accumulated context into a refined answer; an Evaluator computes a confidence score (1–5) against a threshold of 3/5 to decide whether further augmentation is needed, implementing adaptive early-stopping (max 3 iterations).

Action Module — Two agents triggered on low confidence: - Explorer (Intrinsic Med-KA): an LLM decomposes the original question into three hierarchical sub-questions (general observation → anatomical analysis → detailed findings), each answered by the same Med-MLLM, progressively extracting visual detail. - Retriever (Extrinsic Med-KA): an LLM extracts key medical concepts from accumulated context, queries the SPOKE biomedical knowledge graph (42M nodes, 160M edges, 41 databases), filters retrieved subgraph facts by embedding similarity, and injects natural-language medical facts into the reasoning history.

Agents communicate via a centralized, append-only reasoning history \(H_t\). The overall process is formalized as: $\(\hat{a}_t = \Phi_{\text{iterative}}\!\left(I,\, q,\, H_{t-1} \cup \bigcup_{i \in A} h_i^t\right)\)$

Few-shot extension: top-\(K\) in-context examples are selected using a dual-similarity score \(\frac{1}{2}[\text{sim}(T, T_i) + \text{sim}(I, I_i)]\) computed via PubMedCLIP.

Key Contributions

  • Training-free agentic meta-harness that wraps arbitrary Med-MLLMs, separating reasoning orchestration from the underlying vision-language model.
  • Intrinsic Med-KA: clinically-motivated coarse-to-fine hierarchical question decomposition via an Explorer agent.
  • Extrinsic Med-KA: SPOKE knowledge graph retrieval with embedding-filtered fact injection via a Retriever agent.
  • Adaptive Evaluator agent that gates augmentation iterations, preventing over-refinement noise.
  • Dual-similarity few-shot example selection extending the framework to in-context learning settings.
  • Evaluation across eight Med-VQA benchmarks with three different Med-MLLM backbones.

Results

  • LLaVA-Med-v1.5 + AMANDA (zero-shot): +19.36% average over direct inference baseline across eight benchmarks.
  • Med-BLIVA + AMANDA vs. Img2LLM: +6.36%; vs. IdealGPT: +5.42% (zero-shot average).
  • Med-InstructBLIP + AMANDA w/ few-shot: average accuracy 66.87% (+18.10% over baseline); peak gains of +41.70% on OL3I and +33.20% on IU-Xray.
  • LLaVA-Med-v1.5 + AMANDA w/ few-shot: average 62.92% (+20.83% over baseline).
  • AMANDA consistently outperforms or matches IdealGPT (the strongest non-medical agent baseline) across all three backbone models in the zero-shot setting.

Limitations

  • Relies on GPT-4o as the core reasoning engine for all agents; inference cost scales with number of augmentation iterations (up to 3×).
  • Confidence threshold (3/5) and maximum iterations (3) are hand-tuned hyperparameters without ablation showing sensitivity.
  • SPOKE knowledge graph is a static external resource; coverage gaps may affect retrieval quality for rare conditions.
  • Evaluation is limited to Med-MLLMs fine-tuned on medical data; generalization to general-purpose VLMs is relegated to an appendix.
  • Open-ended recall metric and closed-ended accuracy are not always comparable across benchmarks, making cross-task aggregation coarse.

Relevance to Harnesses / Meta-Harnesses

AMANDA is a textbook harness pattern: it wraps an existing model (the Med-MLLM) with an orchestration layer of specialized agents without modifying the wrapped model's weights, making the harness fully model-agnostic. The Evaluator agent functions as a meta-controller that dynamically decides whether to invoke sub-agents, instantiating the core harness concept of adaptive loop control. The architecture demonstrates how a harness can compose multiple retrieval and decomposition strategies (KG-RAG, question decomposition, in-context learning) as pluggable action modules, with a shared reasoning history serving as the inter-agent communication bus. This is directly relevant to harness designers building domain-specialized wrappers over foundation models where the augmentation policy itself (when and what to invoke) is a first-class design concern.