Revisiting Confidence Calibration for Misclassification Detection in VLMs¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Standard calibration degrades misclassification detection in VLMs; key theoretical finding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Standard confidence calibration for VLMs theoretically limits misclassification detection (MisD) performance because aligning a reliability curve to the diagonal does not maximize separation between correct and incorrect predictions. This paper derives a new MisD-oriented calibration objective based on a normalized sigmoid reliability curve, proves it dominates perfect calibration on MisD precision, and implements it via a lightweight post-hoc meta network that predicts per-sample temperature scaling factors.

Problem¶

Confidence calibration (e.g., temperature scaling) is widely applied to VLMs to align predicted confidence with empirical accuracy, but it inherently caps MisD performance: even a perfectly calibrated model assigns expected confidence $\mathbb{E}[s \mid s \geq r]$ as precision for detecting correct predictions—strictly below 1 in realistic distributions. No prior work had theoretically analyzed why calibration fails for MisD or proposed a calibration objective explicitly designed to improve it.

Method¶

Reliability-curve analysis. The paper proves (Lemma 4.1) that for a confidence interval $[a,b]$, the precision of correct-prediction detection equals $\int_a^b w(s)f(s)ds / \int_a^b w(s)ds$, where $f$ is the reliability curve and $w$ is the sample density. Theorem 4.2 shows that perfect calibration ($f(s)=s$) caps this at $\mathbb{E}[s \mid s \geq r] < 1$.

MisD-oriented target curve. The proposed objective maximizes area above the curve in $[0, 0.5]$ and area below in $[0.5, 1]$, solved by a normalized sigmoid: $$\Psi(s) = \frac{\sigma\!\left(\tfrac{s-0.5}{\lambda}\right) - \sigma\!\left(\tfrac{-0.5}{\lambda}\right)}{\sigma\!\left(\tfrac{0.5}{\lambda}\right) - \sigma\!\left(\tfrac{-0.5}{\lambda}\right)}$$ where $\lambda \to \infty$ recovers perfect calibration and $\lambda \to 0$ yields a step function that assigns all correct predictions higher confidence than all incorrect ones (Corollary 4.3). Theorem 4.4 proves $\Psi$ achieves strictly higher MisD precision than the diagonal for any $r \in (0.5,1)$; Theorem 4.5 proves it has lower per-confidence entropy (less mixing).

Surrogate loss. Since $\Psi$ is not directly usable as a loss, the differentiable surrogate $\mathcal{L}_\text{SUR}$ penalizes: (i) high-confidence misclassifications via $\varphi\{\Psi(s),\, -[1/c,\ldots,1/c]^\top \log f(x)\}$ to encourage high-entropy outputs; (ii) low-confidence correct predictions via $\varphi\{1-\Psi(s),\, -y_i^\top \log f(x_i)\}$ to encourage low-entropy (confident) outputs; balanced by hyperparameter $\beta$.

Lightweight meta network (LMN). A post-hoc, parameter-frozen framework trains only small FC layers to predict a per-sample temperature $\tau_v = \sigma^+(\text{FC}_\tau(h_z \| h_x \| h_t))$ from logits $z_v$, image embedding $\xi(x_v)$, and predicted text embedding $\psi(t_p)$. Only the meta network parameters are updated; the underlying VLM is untouched.

Key Contributions¶

First theoretical analysis showing standard calibration inherently limits MisD, via the reliability diagram lens.
New calibration objective (normalized sigmoid curve $\Psi$) with three formal guarantees: (1) interpolates to perfect calibration, (2) dominates calibration precision for both correct/incorrect detection, (3) achieves lower per-confidence mixing entropy.
Differentiable surrogate loss $\mathcal{L}_\text{SUR}$ enabling practical optimization of the non-differentiable curve.
Post-hoc LMN framework preserving zero-shot VLM predictions while yielding instance-specific temperatures.

Results¶

The paper reports experiments on six datasets evaluating MisD performance across multiple metrics. Specific headline numbers from the truncated text are not available, but the paper states: - The proposed method "consistently improves MisD over the recent uncertainty estimation methods across diverse settings." - Baselines include standard temperature scaling (TS), DAC, DOR (VLM-specific calibration methods), FSMisD, and ViLU (VLM MisD methods). - Theoretical precision bounds are validated empirically in Appendix D.8, confirming wide confidence spread in CLIP models that prevents perfect calibration from achieving high MisD.

Limitations¶

$\lambda$ must be tuned; too small causes training instability and overfitting on the calibration set.
The calibration set required to train the LMN may be small in practice, which the surrogate loss partially mitigates but does not eliminate.
The post-hoc framework adds a small network, though inference overhead is not quantified in the provided text.
Experiments focus on CLIP-style contrastive VLMs; generalization to generative VLMs (e.g., LLaVA-style) is not addressed.
The theoretical guarantees assume a fixed threshold $r = 0.5$; behavior at other thresholds is not formally bounded beyond Theorem 4.4's stated range.

Relevance to Vision-Language Models¶

This work directly targets a practical deployment concern for CLIP-based VLMs: even well-calibrated models cannot reliably flag their own mistakes, which matters in safety-critical pipelines (medical, autonomous driving) that use VLMs. The post-hoc, zero-shot-preserving design is well-suited to the typical VLM usage pattern where the backbone is frozen. It advances the line of VLM calibration work (DAC, DOR) by demonstrating that the calibration objective itself—not just the adaptation mechanism—must change to improve MisD, shifting the field's framing from "better calibration → better MisD" to "calibration and MisD require distinct objectives." Researchers tracking VLMs for high-stakes applications should consider this as a drop-in reliability layer on top of any CLIP-based model.