Exploring Hierarchical Consistency and Unbiased Objectness for Open-Vocabulary Object Detection¶

🕒 Published (v1): 2026-04-25 15:06 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Hierarchical consistency with unbiased objectness for CLIP open-vocab detection

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a pseudo-labeling framework for open-vocabulary object detection (OVD) that fixes two failure modes in prior work: CLIP's image-level bias causing 76% of pseudo labels to be background, and RPN objectness scores biased against novel classes. The two components—Hierarchical Confidence Calibration (HCC) and LoCLIP—together establish new state-of-the-art on OV-COCO and OV-LVIS.

Problem¶

Existing pseudo-label-based OVD methods suffer from (1) inaccurate class confidence scores because CLIP is optimized for image-level rather than region-level predictions, resulting in ~76% of COCO pseudo labels being background regions; and (2) unreliable objectness scores from RPNs that were trained exclusively on base classes and therefore suppress novel-class instances as background.

Method¶

Hierarchical Confidence Calibration (HCC): For each candidate region $b$ with visual feature $v_b = V(x_b)$, standard CLIP confidence $\hat{p} = \max(p)$ is calibrated using LLM-generated super- and sub-categories ($K$ per novel class). Sub-category probabilities $p^{\text{sub}} \in \mathbb{R}^{|C_N| \times K}$ are computed over a joint softmax across all $|C_N| \times K$ sub-categories. Max-pooled class-wise scores $z^{\text{sub}}(n) = \max(p^{\text{sub}}(n))$ reweight original class probs: $$r^{\text{sub}}(n) = \frac{p(n)\, z^{\text{sub}}(n)}{\sum_m p(m)\, z^{\text{sub}}(m)}$$ Symmetric calibration with super-categories yields $r^{\text{sup}}$; final score $\hat{r} = \max(r)$ where $r(n) = (r^{\text{sub}}(n) + r^{\text{sup}}(n))/2$. Pseudo labels are assigned only when $\hat{r} \geq \gamma$.

LoCLIP: A parameter-efficient CLIP adaptation (~3K trainable parameters vs. 41M in RPN) that appends a learnable [OBJ] token to CLIP's frozen ViT encoder. The [OBJ] token attends to patch-level features via self-attention; its output is passed through an FC+sigmoid layer to predict objectness. Trained with binary cross-entropy on IoU-based labels using only 1% of training images, converging in ~5 minutes on a single A6000.

Training: Classification loss is weighted by HCC confidence $z_i$ and regression loss by LoCLIP objectness $o_i$ for novel-class pseudo labels, suppressing noisy supervisory signal.

Key Contributions¶

HCC technique: calibrates CLIP region confidence using hierarchical consistency (class/super-/sub-category); proven to boost score when predictions are consistent (Eq. 7) and suppress when inconsistent (Eq. 8)
LoCLIP: parameter-efficient objectness estimator that is class-agnostic and far less biased toward base classes than the RPN
No auxiliary datasets required; pseudo labels used directly without self-training denoising strategies
New state of the art on OV-COCO and OV-LVIS benchmarks

Results¶

Sets new state of the art on OV-COCO ($\text{AP}^N_{50}$ and $\text{AP}^B_{50}$) and OV-LVIS (rare/common/frequent AP) — exact numbers are in truncated sections not provided in the excerpt
LoCLIP converges in ~5 min on a single GPU using only 1% of COCO/LVIS training images
Prior methods generate ~76% background pseudo labels on COCO; HCC substantially reduces this rate (quantitative figures in supplementary)

Limitations¶

HCC relies on LLM-generated hierarchies (e.g., GPT-OSS); quality depends on LLM outputs and may vary across domains with unusual taxonomies
LoCLIP objectness training still uses IoU with base-class objects as supervision, which may retain residual base-class bias
Threshold hyperparameters $\gamma$ and $\tau$ require tuning; supplementary reports sensitivity analysis but adds deployment complexity
Evaluation limited to COCO and LVIS; generalization to more fine-grained or domain-specific OVD benchmarks is unverified

Relevance to Vision-Language Models¶

This work directly targets a fundamental limitation of CLIP for dense prediction tasks: the model was trained image-level but OVD requires region-level reliability. The HCC technique is a principled, training-free recalibration of CLIP scores that exploits semantic structure already encoded in the model, a broadly applicable insight for any VLM-driven region-grounding pipeline. LoCLIP demonstrates that a single learnable token is sufficient to extract objectness information from frozen CLIP features, relevant to the growing literature on parameter-efficient adaptation of VLMs (e.g., prompt tuning, adapter methods) for detection and grounding. The finding that hierarchical consistency of CLIP predictions correlates with region localization quality connects to broader work on probing and exploiting the emergent structure within VLM representations.