Refine and Align: Confidence Calibration through Multi-Agent Interaction in VQA¶

🕒 Published (v1): 2025-11-14 11:08 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AlignVQA is a two-stage multi-agent debate harness for confidence calibration in multiple-choice VQA: diverse specialized VLMs generate candidate answers via distinct prompting strategies, then generalist agents deliberate via structured argument exchange to refine both answers and confidence estimates. Alongside the framework, the authors introduce AlignCal, a differentiable loss that minimizes a tight upper bound on calibration error (UBCE), used to fine-tune the specialist agents before debate. Together, the debate pipeline and AlignCal substantially reduce Expected Calibration Error (ECE) across two medical/scientific VQA benchmarks.

Problem¶

Modern VLMs deployed in high-stakes VQA (medical diagnostics, autonomous navigation) are systematically overconfident — they report high confidence even on wrong answers. Prior calibration methods (temperature scaling, label smoothing, Bayesian fine-tuning) either require retraining with generic objectives, operate single-pass without iterative refinement, or are post-hoc fixes decoupled from answer generation. Multi-agent debate for calibration in the VQA+VLM setting is unexplored.

Method¶

Stage 1 — Agent Ensemble and Stance Clustering. Four specialist VLM agents (Qwen2.5-VL-3B-Instruct, LLaVA-OneVision, Gemma 3 4B, Phi-4-multimodal) each use a distinct prompting strategy (Chain-of-Thought, SelfAsk, Search-style, GENREAD). Each agent independently produces a candidate answer and a confidence estimate (geometric mean of next-token probabilities). Semantically equivalent answers are merged into $K$ stances $\{s_1,\ldots,s_K\}$ via GPT-3.5 as a judge, yielding stance triplets $(s_k, f_k, \bar{c}_k)$ — stance, frequency, mean confidence.

Stage 2 — Generalist Debate and Aggregation. $M$ generalist agents are instantiated with stances sampled proportionally to $f_k$, maintaining soft majority bias while allowing minority exploration. Each agent develops supporting rationales, receives one supporting and one opposing argument pair (factuality-checked via Chain-of-Verification + search augmentation), then outputs a revised answer $y_j'$ and refined confidence $\text{Conf}(y_j')$. Final answer is the majority-vote stance; final confidence is the mean refined confidence of agents supporting that stance.

AlignCal Loss. A differentiable surrogate for UBCE (Upper Bound Calibration Error), defined per-example as: $$L_{\text{AlignCal}}(p_y, p_{\max}) = p_y(1 - p_{\max}) + (1 - p_y)p_{\max}$$ where $p_y$ is the softmax probability of the true class (plug-in for $q(x) = \mathbb{E}[t|x]$) and $p_{\max}$ is the top confidence. Minimizing $\mathbb{E}[L_{\text{AlignCal}}] \approx \text{UBCE}$ directly reduces ECE by design. Full objective: $L_{\text{tot}} = L_{\text{FL}} + \lambda L_{\text{AlignCal}}$.

Key Contributions¶

AlignVQA framework: two-stage debate harness (specialist ensemble → generalist deliberation with argument exchange) for post-hoc calibration in MCQ VQA.
AlignCal loss: differentiable surrogate to UBCE derived via the plug-in principle; first calibration loss with an explicit probabilistic upper-bound interpretation for VQA.
Stance clustering with semantic deduplication: GPT-3.5 judge merges lexically diverse but semantically equivalent answers before debate, preventing fragmentation of majority support.
Calibrated specialists → better debate outcomes: shows that AlignCal fine-tuning of specialist agents provides additive gains when combined with the debate framework.

Results¶

VQARad (medical VQA): agentic framework alone reduces ECE from 0.375 (Gemma 3 4B baseline) to 0.146, ACE from 0.207 to 0.133; combined Agentic+AlignCal+FL achieves ECE 0.098, ACE 0.095.
ScienceQA: AlignCal fine-tuning alone reduces ECE from 0.232 (focal-loss Gemma 3 4B) to 0.058; Agentic+AlignCal+FL achieves ECE 0.055, ACE 0.110 — best across all calibration strategies tested.
Outperforms temperature scaling (TS), Dirichlet calibration (DC), focal loss (FL), and label smoothing (LS) on ECE and ACE across both datasets.
Accuracy is competitive: VQARad 68.2% (vs. best baseline 69.0%), ScienceQA 76.1% (vs. best baseline 76.0%).

Limitations¶

Per-question inference cost scales as $O(N \cdot M)$ VLM calls plus external GPT-3.5 calls for stance merging.
AlignCal does not guarantee worst-case (MCE) improvements without additional uniformity constraints; empirical gains observed but not proven.
Evaluated only on two MCQ datasets; open-ended VQA or free-form generation settings not addressed.
Dirichlet calibration comparison unavailable on ScienceQA due to class-probability inaccessibility, making one baseline comparison incomplete.
Semantic clustering depends on GPT-3.5 as an oracle judge, introducing an external dependency and cost.

Relevance to Harnesses / Meta-Harnesses¶

AlignVQA is a concrete instantiation of a calibration meta-harness: rather than improving a single model, it orchestrates heterogeneous specialist agents with defined roles, argument-passing protocols, and aggregation rules to produce a higher-quality system-level output (calibrated confidence). The two-stage design — specialist generation followed by generalist deliberation — is a reusable harness pattern directly applicable to any ensemble VLM pipeline where calibration quality matters. Notably, it illustrates the coupling between agent-level training (AlignCal) and harness-level aggregation design: training specialists explicitly for harness participation improves the downstream debate outcome, a key design principle for meta-harness builders. The use of Chain-of-Verification and search-augmented factuality checking within the deliberation stage demonstrates how tool-augmented sub-agents can be embedded inside a larger orchestration loop.