Measuring and Improving Complex-Atomic Answer Consistency in Endoscopic VQA

🕒 Published (v1): 2026-07-20 11:25 UTC · Source: Arxiv · link

Why this paper was selected

Complex-atomic answer consistency metric for endoscopic VQA; fills medical eval gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Endoscopic VQA models can produce correct answers to complex multi-component questions while simultaneously failing the simpler atomic sub-questions that constitute those answers — a hidden inconsistency that final-answer accuracy metrics cannot detect. This paper introduces EndoCA, a paired complex-atomic benchmark built from Kvasir-VQA and Kvasir-VQA-x1, and Atomic-Support Reconciliation (ASR), a training-free inference-time mechanism that uses model-generated atomic answers as premises for answer revision and selective abstention.

Problem

Existing endoscopic VQA benchmarks report only final complex-answer accuracy, which can be high even when the same model fails the atomic sub-questions underlying that answer on the same image. This complex-atomic inconsistency masks unreliable visual interpretation in a clinically sensitive domain, and no prior benchmark directly measures it.

Method

EndoCA benchmark. Paired evaluation units are constructed from Kvasir-VQA (atomic QA) and Kvasir-VQA-x1 (complex QA over the same images). Question complexity \(m_i\) is the number of associated atomic questions for complex question \(Q_i\). Four metrics are reported:

\[\text{Complex-Atomic Inconsistency} = \frac{\sum_{i=1}^N C_i(1 - A^*_i)}{\sum_{i=1}^N C_i}\]

where \(C_i \in \{0,1\}\) is complex-answer correctness and \(A^*_i = \prod_{j=1}^{m_i} A_{ij}\) requires all atomic answers to be correct. EndoCA-Core (27,736 QA items/model; complexity 1–2) is the primary suite; EndoCA-Diagnostic (15,300 QA items/model; adds complexity-3 stratum) supports controlled analysis.

Atomic-Support Reconciliation (ASR). Training-free, three-step inference pipeline: 1. Direct answer: model answers complex question \(Q_i\) → \(\hat{A}_i\). 2. Atomic answers: model answers each \(q_{ij}\) → \(\hat{a}_{ij}\); these form atomic premise set \(P_i = \{(q_{ij}, \hat{a}_{ij})\}_{j=1}^{m_i}\). 3. ASR-Revise: model receives image, \(Q_i\), \(\hat{A}_i\), and \(P_i\) as context and produces reconciled answer \(\tilde{A}_i\).
ASR-Selective: after revision, model judges whether \(\tilde{A}_i\) is consistent with \(P_i\) and outputs answer/abstain; abstentions are excluded from accuracy computation.

All prompts are fixed templates; no additional training or LLM judge is used.

Key Contributions

• EndoCA benchmark: paired complex-atomic evaluation suite with EndoCA-Core and EndoCA-Diagnostic suites covering 11 evaluated VLMs across four model groups.
• Empirical finding: complex-answer accuracy systematically overstates model reliability; rankings shift substantially when joint accuracy (both complex and all atomic correct) is required.
• ASR: training-free inference mechanism using atomic premises for answer revision (ASR-Revise) and consistency-guided selective abstention (ASR-Selective).
• Ablation: atomic premises are load-bearing — self-revision without them fails to improve joint accuracy or reduce inconsistency.

Results

• MedGemma-FT vs. Qwen2.5-VL-FT (EndoCA-Core): near-identical complex accuracy (74.5% vs. 72.1%), yet joint accuracy differs by 14.2 pp (50.7% vs. 36.8%); illustrates that complex-answer accuracy alone is uninformative about paired reliability.
• LLaVA-Med: complex accuracy 28.7%, atomic accuracy 17.0%, complex-atomic inconsistency 83.1% — the worst inconsistency among evaluated models.
• GPT-5.5 (closed-source): 60.8% complex, 55.8% atomic, 41.7% joint, 24.4% inconsistency — strongest overall profile.
• ASR-Revise (averaged over 4 models on EndoCA-Core): joint accuracy +5.4 pp; complex-atomic inconsistency reduced by 10.4 pp on average; individual gains +3.1 to +8.1 pp joint.
• ASR-Selective: 75.9% average coverage, 71.0% selective complex accuracy (vs. 63.3% direct full-coverage accuracy).
• Ablation (Self-Revise without atomic premises): joint accuracy 43.0% and inconsistency 28.2%, worse than ASR (50.9%, 17.1%), confirming atomic premises are essential.

Limitations

• ASR introduces additional inference calls per sample (one complex + \(m_i\) atomic + one revise/selective), increasing latency proportionally to question complexity.
• ASR-Selective reduces coverage to ~76% on average; abstained samples are simply excluded, with no fallback or escalation mechanism.
• Benchmark built from one gastrointestinal endoscopy dataset pair (Kvasir-VQA / Kvasir-VQA-x1); generalizability to other endoscopic modalities (capsule endoscopy, bronchoscopy) is untested.
• Atomic answer quality is limited by the underlying model; if atomic predictions are systematically wrong, ASR uses flawed premises for revision.
• Complex-answer scoring requires all associated atomic annotations to match, a conservative criterion that may penalize valid paraphrases.

Relevance to Vision-Language Models

EndoCA directly benchmarks a structural failure mode of VLMs — the gap between holistic multi-component question answering and fine-grained atomic visual grounding — which is orthogonal to standard accuracy metrics and relevant to any VLM evaluation in compositional or multi-fact settings. The ASR mechanism is a prompt-only, model-agnostic consistency enforcement strategy applicable to any autoregressive VLM without fine-tuning, making it directly transferable to other medical or general-purpose VQA pipelines. For researchers tracking VLMs, the work demonstrates that complex-answer benchmarks must be complemented by sub-question consistency checks to avoid rewarding surface-level answer matching over genuine visual comprehension. The evaluation of 11 VLMs including frontier closed-source models (GPT-5.5, Grok-4.20, Claude Opus 4.7) and endoscopy-adapted LoRA models provides a comparative cross-group reference point for VLM capability in the medical vision domain.