Skip to content

AQuA: Toward Strategic Response Generation for Ambiguous Visual Questions

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Strategic response generation for ambiguous VQA; addresses real-world uncertainty

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AQuA is a 7.2K-sample VQA dataset that classifies ambiguous image–question pairs into four fine-grained levels and pairs each with a ground-truth response strategy (direct answer, contextual inference, listing alternatives, or clarification request). Evaluations reveal that all current VLMs—including GPT-5 and Gemini 2.5 Flash—default to overconfident single answers and fail strategically on ambiguous levels. Fine-tuning small (~2–3B) models on AQuA via SFT+GRPO raises overall strategic accuracy to ~80%, outperforming all untuned baselines including 78B-parameter open-source models.

Problem

Standard VQA benchmarks contain unambiguous image–question pairs, leaving untested a critical real-world capability: adapting response strategy to the degree and nature of ambiguity. Prior work either ignores ambiguity entirely or collapses it to a binary answer-or-ask decision (e.g., ClearVQA). No resource existed for systematic, multi-level training or evaluation of strategy selection in visual QA.

Method

Dataset construction. Images are drawn from COCO; object saliency is quantified as a weighted combination of area ratio (weight 0.7) and normalized distance to image center (weight 0.3). Ambiguity level is assigned by the number of high-saliency objects (score > 0.6): zero → Level 0, one → Level 1, two–three → Level 2, five+ → Level 3. Question–answer pairs are generated by GPT-5 with level-specific prompts, then filtered by GPT-5-mini in three stages (level conformity, cross-level uniqueness, factual correctness) and human-validated via MTurk (dual-annotator PASS/FAIL).

Training pipeline. Models (Qwen2.5-VL-3B-Instruct, InternVL3-2B-Instruct) are first trained with supervised fine-tuning (SFT) on the 3.6K training split to teach the strategy space, then refined with Group Relative Policy Optimization (GRPO). The GRPO reward is:

\[R(y \mid x, I) = \begin{cases} 1 - \lambda & \text{strategy correct, factual distortion detected} \\ 1 & \text{strategy correct, no distortion} \\ 0 & \text{otherwise} \end{cases}\]

with \(\lambda = 0.3\). A GPT-5-mini LLM-as-judge assigns rewards; automatic judge agreement with human evaluation is 98.5%.

Key Contributions

  • AQuA: first VQA dataset with four fine-grained ambiguity levels, each annotated with the contextually optimal response strategy; 7.2K samples balanced across levels (900/level/split), human-validated.
  • Systematic empirical analysis showing that neither model scale (3B→78B), chain-of-thought prompting, nor explicit strategy prompting reliably resolves the strategic failure modes.
  • SFT+GRPO training recipe enabling small open-source VLMs to reach ~80% overall strategic accuracy, surpassing GPT-5 and Gemini 2.5 Flash.

Results

  • Base models (zero-shot): Level 0 strategic accuracy is high (89–99% across all models), but Levels 1–3 collapse to ~0–33%, with overall strategic accuracy in the 22–33% range for all tested models (Qwen2.5-VL-3B: 32.83%; GPT-5: 22.86%).
  • Chain-of-Thought: No meaningful improvement; models produce verbose single-answer outputs (GPT-5 overall: 25.72% vs. 22.86% zero-shot).
  • Strategy prompting: Modest gains for stronger models only (GPT-5 rises to 42.25% overall; Gemini 2.5 Flash to 36.98%); small open-source models are unaffected.
  • AQuA-tuned models: Qwen2.5-VL-3B-Tuned achieves Level 0/1/2/3 strategic accuracy of ~99.56/77.0/82.22/86.33%; InternVL3-2B-Tuned ~80%+ across levels—both outperform all untuned baselines including 72B/78B variants.
  • Human evaluation (200 examples): Level 0/1 achieve near-perfect human agreement (50/50 and 48/50); Levels 2–3 show 32/50 agreement, reflecting inherent subjectivity at the list-vs-clarify boundary.

Limitations

  • AQuA is fully synthetic (GPT-5-generated QA pairs on COCO images), which may not capture the full diversity of real-world ambiguous queries.
  • Saliency scoring via area ratio and image-center distance is a coarse proxy; edge cases (occluded objects, non-central salient items) may be mislabeled.
  • The strategy taxonomy is heuristic; borderline cases between Level 2 and Level 3 show ~36% human disagreement.
  • GRPO uses a small sample (15 instances/level for training) and relies on GPT-5-mini as judge, coupling evaluation reliability to a proprietary model.
  • Only 2–3B-parameter models are fine-tuned; it is unclear whether the approach scales or generalizes to other architectures.

Relevance to Vision-Language Models

AQuA directly targets a gap in VLM evaluation: existing benchmarks do not stress-test adaptive uncertainty handling, a prerequisite for safe deployment in open-ended multimodal dialogue. The finding that scale alone (3B→78B, including GPT-5) fails to confer strategic ambiguity awareness challenges the common assumption that capability improvements transfer to pragmatic response selection. The SFT+GRPO recipe with LLM-as-judge rewards offers a transferable training paradigm for instilling meta-cognitive response strategies into VLMs, relevant to any work on calibrated or uncertainty-aware multimodal systems.