Skip to content

Confidence Scores in Open-Vocabulary Detection Are a Biased Mixture of Scale and Semantics

🕒 Published (v1): 2026-07-13 01:33 UTC · Source: Arxiv · link

Why this paper was selected

Open-vocab detector confidence = scale+semantics mixture; exposes fundamental reliability flaw in CLIP-based OVD

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Open-vocabulary detectors (GroundingDINO, OWL-ViT, YOLO-World) produce confidence scores \(s = \cos(v, t)\) that are structurally biased mixtures of object scale and query specificity, not calibrated localization probabilities. The paper formally decomposes this as \(s \approx \alpha \cdot \phi(a) + \beta \cdot \psi(d_\text{sem}) + \varepsilon\) and shows both biases are theoretically inevitable from CLIP's image-level pretraining. A parameter-free temperature-scaling correction recovers 19.6% small-object Recall@10 at a measurable precision cost.

Problem

CLIP-based open-vocabulary detectors output cosine similarity as a scalar confidence score, and practitioners use a fixed threshold to filter detections under the implicit assumption that the score measures localization certainty (\(s \approx P(\text{IoU} \geq 0.5 \mid v, t)\)). No prior work had identified or quantified the structural sources of miscalibration: (1) scores are inflated for large objects and (2) suppressed for generic text queries, independent of true localization quality.

Method

The paper proceeds in three steps:

  1. Theoretical decomposition. Using a signal-plus-noise model for region features, \(v_\text{raw} = \mu + \bar{\eta}(R)\), a second-order expansion of \(\cos(\theta + \delta)\) yields: $\(E[s \mid a] \approx \cos\theta \left(1 - \frac{\kappa}{2a}\right)\)$ showing scale bias is an angular-concentration effect of spatial pooling. Semantic bias follows from CLIP's contrastive geometry: specific queries occupy tighter embedding clusters than generic ones.

  2. Controlled experiments on COCO (8 categories, 640 image–category pairs). Three experiments: (Exp. 1) scale bias with query fixed; (Exp. 2) semantic bias with visual content and bounding box fixed, varying query specificity across a four-level hierarchy (e.g., "a dog" → "a pet" → "an animal" → "an object"); (Exp. 3) 2×2 factorial crossing scale × query specificity. Scale bias is replicated on LVIS v1 val (1,203 categories, \(n=1{,}816\) annotations).

  3. Parameter-free test-time correction. Variance-matched temperature scaling \(s_\text{corr} = s^{1/T_g}\) where \(T_g = \sqrt{\text{Var}(\log s_g) / \text{Var}(\log s_\text{large})}\), estimated without ground-truth labels (\(T_\text{small}=1.28\), \(T_\text{medium}=1.06\), \(T_\text{large}=1.00\)).

Key Contributions

  • Formal additive decomposition \(s \approx \hat{\alpha}\cdot\phi(a) + \hat{\beta}\cdot\psi(d_\text{sem}) + \varepsilon\) with quantified coefficients (\(\hat{\alpha}=+0.064\), \(\hat{\beta}=-0.705\), both \(p<10^{-40}\)) and structural derivation showing both biases are unavoidable from image-level pretraining.
  • Cross-architecture validation of scale bias across GroundingDINO, OWL-ViT, and YOLO-World (\(r \in [0.36, 0.64]\), all \(p < 10^{-20}\)), replicated on LVIS 1,203 categories.
  • Demonstration that oracle per-scale thresholding cannot recover small-object localization signal (\(\Delta F1 = +0.001\) for small vs. \(+0.102\) for large), proving the bias is not fixable by threshold tuning.
  • 2×2 factorial experiment revealing a floor effect: scale bias compresses small-object scores to near zero, suppressing the semantic signal (semantic \(\Delta\) is 2.2× larger for large than small objects).
  • Parameter-free temperature-scaling correction with measured precision/calibration trade-offs (no architectural changes or retraining required).

Results

  • Scale bias (COCO, GroundingDINO): mean confidence 0.180 (small) → 0.317 (medium) → 0.520 (large); \(r=0.579\), \(p=1.29\times10^{-58}\); ANOVA \(F=146.92\).
  • Scale bias cross-architecture: OWL-ViT \(r=0.363\); YOLO-World \(r=0.637\); all \(p<10^{-20}\).
  • Scale bias on LVIS (1,203 categories): \(r=0.369\), \(p=1.08\times10^{-59}\); rare categories show strongest bias (\(r=0.491\)) vs. frequent (\(r=0.386\)).
  • Semantic bias: confidence drops from L0 (specific, mean 0.403) to L3 (generic, mean 0.194); paired \(t=16.5\), \(p=9.83\times10^{-49}\).
  • AP gap by scale (GroundingDINO): small AP=0.430, medium=0.566, large=0.704; gap of 0.274.
  • Threshold asymmetry: achieving 50% recall for small objects requires threshold 0.129 vs. 0.476 for large (3.7× lower).
  • Calibration gap: small objects \(-0.384\) vs. large \(-0.205\) (under-confident at all scales).
  • Temperature-scaling correction: small-object Recall@10 +19.6% (\(p<0.01\)); overall Recall@10 +4.1% (\(p<0.05\)); small-object ECE halved (0.102→0.047); pooled AP decreases by 0.025 (\(p=0.002\)).

Limitations

  • Ideal score \(s^*\) is defined solely as localization correctness (\(\text{IoU}\geq0.5\)); semantic grounding uncertainty is a distinct, unaddressed axis.
  • Potential confounds (occlusion, background clutter, proposal quality, category frequency) not fully disentangled from the pure scale mechanism; the occlusion confound is ruled out (\(r=-0.000\), \(p=0.99\)) but others remain.
  • Semantic bias experiments conducted only on GroundingDINO over 8 COCO categories; cross-detector replication not provided for semantic bias.
  • All main controlled experiments use COCO 2014 val; LVIS replication covers scale bias only; broader dataset/architecture validation is future work.
  • Temperature-scaling correction evaluated on \(n=79\) mixed-scale pairs; generalization across detectors and downstream tasks not demonstrated.
  • Which architectural component (spatial pooling vs. contrastive alignment objective) drives each bias is not disentangled.

Relevance to Vision-Language Models

This paper directly implicates CLIP's image-level contrastive pretraining as the structural cause of confidence miscalibration in region-level tasks — a foundational concern for any VLM applied to dense prediction. It quantifies a gap that has practical consequences for VLM-based pipelines in robotics, surveillance, and medical imaging, where small/rare objects are critical. The additive decomposition \(s \approx \alpha\phi(a)+\beta\psi(d_\text{sem})+\varepsilon\) provides a diagnostic framework applicable to any architecture sharing the \(\cos(v,t)\) confidence interface (GLIP, RegionCLIP, OWL-ViT v2, etc.). The finding that prompt wording ("a dog" vs. "an animal") produces a quantifiable, statistically significant score shift is directly relevant to the prompt-sensitivity literature for VLMs in zero-shot deployment.