Skip to content

ProCal: Inference-Time Proposal Calibration for Open-Vocabulary Object Detection

🕒 Published (v1): 2026-07-02 06:18 UTC · Source: Arxiv · link

Why this paper was selected

Inference-time calibration for open-vocab detection; plug-in improvement without retraining

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ProCal is a training-free, plug-in inference-time module that recalibrates novel-category detection scores in open-vocabulary object detectors by injecting foreground/background localization cues from a frozen VLM. It addresses the systematic ranking bias where novel-object proposals score below background proposals due to base-class-only detector training. Applied to CLIPSelf ViT-L/14, it yields +2.5 APr on OV-LVIS with no additional training.

Problem

F-VLM-style open-vocabulary detectors freeze a pretrained VLM as the image backbone and combine its cosine-similarity scores with detector confidence. Because the detector is trained exclusively on base-category annotations, proposals containing novel objects are routinely assigned low confidence or suppressed as background, causing novel proposals to be ranked below true background proposals at inference time. Existing remedies (retraining, distillation, unknown-object supervision) require modifying the detector pipeline.

Method

ProCal operates post-hoc at inference on the top-\(k\) proposals from the existing detector's RPN/RCNN pipeline:

  1. Proposal feature extraction. Each proposal region \(I_{b_i}\) is cropped and encoded by a separate frozen VLM image encoder (OpenCLIP ViT-B/16): $\(v_i = \frac{E_{\text{img}}(I_{b_i})}{\|E_{\text{img}}(I_{b_i})\|_2}\)$

  2. Three class-agnostic prompts are encoded:

  3. \(p_1\): "a photo of an object in the background"
  4. \(p_2\): "a photo of an object"

  5. \(p_3\): "a photo of a background"

  6. Localization-aware foreground score exploits the contrast between \(p_1\) and \(p_2\) to highlight object-containing regions: $\(s_i^{\text{loc}} = \varphi(v_i, t_1) - \varphi(v_i, t_2)\)$

  7. Background-aware suppression score measures background resemblance: $\(s_i^{\text{bg}} = \varphi(v_i, t_3)\)$

  8. Proposal prior maps both scores through sigmoid with temperatures \(\tau_1, \tau_2\) and combines them: $\(q_i = \lambda\,\sigma(s_i^{\text{loc}}/\tau_1) + (1-\lambda)\,\sigma(-s_i^{\text{bg}}/\tau_2)\)$

  9. Score fusion applies recalibration only to novel categories, leaving base scores unchanged: $\(\tilde{P}_i^j = \begin{cases} P_i^j & j \in C_{\text{base}} \\ P_i^{j\,\gamma}\,(q_i)^{1-\gamma} & j \in C_{\text{novel}} \end{cases}\)$ Default hyperparameters: \(\lambda=0.5\), \(\tau_1=0.02\), \(\tau_2=0.1\), \(\gamma=0.2\).

Key Contributions

  • Empirical observation that frozen VLMs encode class-agnostic foreground/background discriminative cues accessible via generic prompts.
  • ProCal: a training-free, architecture-agnostic plug-in that constructs a proposal prior from localization-aware and background-aware scores.
  • Analysis showing that ProCal improves proposal-level ranking quality: increased fraction of novel-positive proposals in top-\(k\) and decreased fraction of background proposals.

Results

  • OV-LVIS (ViT-L/14): CLIPSelf + F-ViT + ProCal achieves APr = 39.3 (+2.5 over CLIPSelf + F-ViT baseline of 36.8); AP = 37.3 (matching OV-DQUO at 37.3).
  • OV-LVIS (ViT-B/16): APr = 26.9 (+1.1 over CLIPSelf + F-ViT baseline of 25.8).
  • OV-COCO (ViT-B/16): \(\text{AP50}_{\text{novel}}\) = 40.1 (+2.7 over CLIPSelf + F-ViT at 37.4); \(\text{AP50:95}_{\text{novel}}\) = 19.3 (+1.4).
  • OV-COCO (ViT-L/14): \(\text{AP50}_{\text{novel}}\) = 45.7 (+1.2 over CLIPSelf + F-ViT at 44.5).
  • Gains are consistent across IoU thresholds (\(\text{AP}^{75}_{\text{Novel}}\) +2.5, \(\text{AP}^{90}_{\text{Novel}}\) +0.2 on OV-COCO ViT-B/16).
  • Most pronounced improvement for medium-sized objects (\(\text{AP}^{\text{Medium}}_{\text{Novel}}\) +2.6).
  • Proposal ranking analysis: background proposal fraction in top-10 drops from 9.5% to 8.0%; novel-positive fraction increases from 25.8% to 24.7% (sic—Table IV shows slight improvement with ProCal).

Limitations

  • Adds inference-time compute overhead: requires a second frozen VLM (OpenCLIP ViT-B/16) to extract per-proposal features via image cropping and re-encoding for every proposal.
  • Hyperparameters (\(\lambda, \tau_1, \tau_2, \gamma\), NMS threshold) are set differently per benchmark (OV-COCO vs. OV-LVIS) and require tuning.
  • Gains on OV-COCO ViT-L/14 are modest (+1.2 APr_novel) and do not surpass NoOVD on that split.
  • Effectiveness is concentrated on medium-scale objects; small and large novel objects see marginal improvement.
  • The method only recalibrates ranking of existing proposals; it cannot recover novel objects whose proposals were entirely suppressed before top-\(k\) selection.

Relevance to Vision-Language Models

ProCal directly exploits CLIP-style VLMs' latent foreground/background discrimination—a property of pretrained image-text alignment not typically leveraged at the proposal level—demonstrating that frozen VLMs encode richer spatial semantics than their intended image-level use. This connects to a growing body of work on transferring VLM representations to region-level tasks (F-VLM, CLIPSelf, CORA) and specifically highlights prompt engineering as a tool for eliciting localization signals from models trained only with image-text contrastive objectives. For VLM researchers, the result that a simple prompt-contrast trick (\(p_1 - p_2\)) reliably identifies object-containing regions is a practically useful insight about the geometry of CLIP embedding space. The training-free, plug-in design also offers a template for post-hoc score calibration in any VLM-backbone detector.