Skip to content

Fantastic Tractor-Dogs and How Not to Find Them With Open-Vocabulary Detectors

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

OVDs produce confident false positives on negative images; critical deployment flaw

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Open-Vocabulary Detectors (OVDs) with early-fusion architectures generate confident false positives on background images (images containing none of the queried classes), a failure mode invisible to standard benchmarks like COCO and LVIS. The root cause is cross-modal attention in vision-language fusion layers, which cannot output zero attention when no prompt token matches. Appending semantically neutral attention sink tokens to the prompt, requiring no retraining, redirects spurious attention and recovers up to \(5\times\) AP at low false positive rates.

Problem

Standard OVD benchmarks (COCO, LVIS) almost never contain images without at least one annotated ground-truth class, masking a pervasive background false positive problem. Early-fusion models (Grounding DINO, LLMDet, GLIP, etc.) exhibit a high background false positive rate (\(\text{FPR}_\text{bg}\)) because their cross-modal attention mechanism must always attend to some prompt token—there is no "none of the above" option. When the queried object is absent, the model distributes the irrelevant class signal across all visual tokens, and the most prevalent signal gets a high confidence score.

Method

The paper proposes appending attention sink tokens to the OVD text prompt as a training-free intervention:

  1. Diagnosis. \(\text{APFPR}_\text{fpr}\) is defined as AP measured after raising the confidence threshold until \(\text{FPR}_\text{bg} \leq \text{fpr}\). LVIS negative labels (classes confirmed absent per image) enable measuring \(\text{FPR}_\text{bg}\) directly.

  2. Sink tokens. One or more semantically neutral token embeddings \(s_i\) are appended to the prompt. Three initialization strategies are explored:

  3. Random: \(s_i \sim \mathcal{N}(0, \sigma^2 I)\)
  4. Mean of vocabulary embeddings: \(s_i = \frac{1}{|V|}\sum_{w \in V} e_w\)

  5. Special-token embedding: \(s_i = e_{[()]}\)

  6. Mechanism. Sink tokens absorb spurious cross-modal attention from visual tokens that correspond to absent prompt classes; predictions whose top class is a sink token are discarded. No fine-tuning is required; sinks are appended at inference time.

  7. Calibration. Optimal sink count and initialization are selected using only 64 VOC images, which strongly predicts LVIS MiniVal performance.

Key Contributions

  • Identifies the systematic \(\text{FPR}_\text{bg}\) failure in all tested early-fusion OVDs and explains why standard benchmarks mask it.
  • Proposes an extension to existing benchmarks (\(\text{APFPR}_\text{fpr}\) metric, LVIS negative-label protocol, POPE repurposing) to quantify background false positives.
  • Demonstrates that late-interaction models (CLIP-style) do not exhibit elevated \(\text{FPR}_\text{bg}\), isolating the cause to cross-modal fusion layers.
  • Provides a training-free attention sink fix applicable out-of-the-box to six early-fusion models.
  • Shows that 64 VOC images suffice to select sink hyperparameters that transfer to 1203-class LVIS.

Results

  • LLMDet Swin-T: \(\text{APFPR}_{0.01}\) increases from 0.045 → 0.223 (+\(4.96\times\)); \(\text{APFPR}_{0.05}\) from 0.140 → 0.326 (+\(2.33\times\)).
  • LLMDet Swin-B: \(\text{APFPR}_{0.05}\) increases from 0.125 → 0.349 (+\(2.79\times\)); full AP improves slightly from 0.466 → 0.499.
  • MM-Grounding DINO Swin-T: \(\text{APFPR}_{0.05}\) from 0.148 → 0.306 (+0.158 absolute).
  • GLIP Swin-T: \(\text{APFPR}_{0.01}\) from 0.116 → 0.166 (+0.050); full AP drops slightly (0.333 → 0.302).
  • Late-interaction models (YOLO-World-L, OV-DINO Swin-T) are unaffected by \(\text{FPR}_\text{bg}\) and serve as the upper-bound reference; e.g., OV-DINO achieves \(\text{APFPR}_{0.01} = 0.274\) without any modification.
  • Residual false positives after sinks resemble late-interaction errors (fine-grained confusion, visual similarity), not semantic hallucination.

Limitations

  • No single sink initialization works across all models; per-model selection is required (though 64 VOC images suffice for this search).
  • Some models (GLIP, GDINO) show a slight full-AP regression when sinks are added due to low-confidence true positives falling below the recalibrated threshold.
  • Evaluation is restricted to LVIS and POPE; results may not generalize to highly domain-specific distributions (security, medical imaging) beyond what is implied.
  • Training on negative examples (Grounding DINO 1.5 approach) still leaves residual \(\text{FPR}_\text{bg}\), and the paper does not directly compare to such trained models at equal scale.
  • The attention sink mechanism is described for early-fusion OVDs; applicability to LLM-backbone detectors with fundamentally different attention structures is not fully analyzed.

Relevance to Vision-Language Models

This paper directly addresses a structural flaw in cross-modal attention—the workhorse of early-fusion VLMs—showing that attention cannot naturally express "no relevant class," causing hallucination-like false positives at inference. The \(\text{FPR}_\text{bg}\) metric and LVIS negative-label protocol provide a transferable evaluation scaffold for any VLM that performs dense grounding. The attention sink solution is closely related to register tokens in vision transformers and attention sink research in LLMs, unifying these threads for the grounding setting. For researchers tracking VLMs, this work is a concrete case study on how benchmark design (positive-only images) systematically hides failure modes that matter in deployment.