Skip to content

GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GUIDED is a three-stage decomposition framework for fine-grained open-vocabulary object detection (FG-OVD) that separates subject localization from attribute discrimination to overcome semantic entanglement in CLIP embeddings. By routing coarse-grained subjects to a detection transformer and fine-grained attribute matching to a refined VLM, it achieves a 23.2% mAP improvement over its LaMI-DETR baseline on FG-OVD and sets new state-of-the-art on both FG-OVD and 3F-OVD benchmarks.

Problem

Existing open-vocabulary detectors encode the full fine-grained class name (e.g., "a dark pink pillow made of fabric") into a single CLIP text embedding. CLIP's contrastive objective treats all tokens equally, causing two failure modes: (1) attribute over-representation, where descriptive modifiers dominate the embedding and shift localization to sub-parts (e.g., detecting "head" instead of "dog" for the query "a dog with a head"); and (2) semantic drift, where fine-grained variants of the same object land far apart in embedding space, making classification unreliable. Both stem from forcing a single embedding to simultaneously serve localization and attribute recognition.

Method

GUIDED decomposes FG-OVD into three explicit stages:

  1. Subject Identification: A frozen LLM (GPT-4o or LLaMA-3.3-70B) parses each fine-grained class name into a coarse-grained subject noun and a list of descriptive attributes. Both are encoded separately via the frozen CLIP text encoder, producing subject embeddings {t_i} and attribute embeddings {t^j_i}.

  2. Coarse-Grained Object Detection (CGOD): Built on LaMI-DETR (DINO + ConvNeXt-Large backbone). Object queries are initialized via top-k selection over coarse subject classification logits. An Attribute Embedding Fusion (AEF) module injects attribute information into these queries via cross-attention, using (attribute_emb − subject_emb) as keys and the union of subject+attribute embeddings as values. The detector outputs bounding boxes and coarse confidence scores s_coarse using only subject embeddings as classifier weights.

  3. Fine-Grained Attribute Discrimination (FGAD): Candidate box features are extracted by RoI-pooling over the frozen ConvNeXt feature map F_conv. A learned lightweight linear projection head is appended to the CLIP text encoder (producing T′) and applied to the full fine-grained class names. Attribute similarity s_fine is computed as softmax cosine similarity between the pooled region feature and projected text embeddings. Final score: s_final = (s_coarse)^α · (s_fine)^(1−α), with α=0.6.

Training is two-stage: (1) pretrain the detector on LVIS base classes for 85,200 iterations; (2) fine-tune for 2,000 iterations on FG-OVD using ground truth labels mapped to coarse subjects, with a binary cross-entropy loss over the combined final score.

Key Contributions

  • Three-stage decomposition (identify → detect → discriminate) that decouples localization from attribute recognition, aligning each subtask to the model best suited for it.
  • Attribute Embedding Fusion (AEF) module that selectively injects attribute cues into DETR object queries via cross-attention on difference vectors (t^attr − t^subject), mitigating over-representation.
  • Projection-based FGAD that adds a learnable linear head atop frozen CLIP to improve fine-grained text-region alignment without disrupting the detection backbone.
  • Model-agnostic design: GUIDED improves Grounding DINO (+12.4%), OWL-ViT (+15.4%), and LaMI-DETR (+23.2%) on FG-OVD with the same framework.
  • LLM robustness analysis showing GPT-4o and LLaMA-3.3-70B both achieve near-perfect subject extraction (≤1/300 critical errors), and the mAP drop with open-source LLaMA-3.1-8B is only 0.5%.

Results

  • FG-OVD (average mAP): GUIDED+LaMI-DETR achieves 66.4% vs. LaMI-DETR baseline 43.2% (+23.2 pp); vs. HA-FGOVD+LaMI-DETR 46.6% (+19.8 pp); vs. best prior competitor OWLv2-L/14 37.3%.
  • FG-OVD per-detector gains: Grounding DINO 33.5%→45.9% (+12.4 pp); OWL-ViT L/14 41.5%→56.9% (+15.4 pp).
  • 3F-OVD (zero-shot transfer, no 3F-OVD training):
  • NEU-171K-C (vehicles): LaMI-DETR 9.0×10⁻⁴ → GUIDED 7.2×10⁻³ (~8× gain).
  • NEU-171K-RP (retail): LaMI-DETR 2.3×10⁻¹ → GUIDED 2.7×10⁻¹.
  • Ablation (FG-OVD mAP): AEF alone adds +4.0 pp; CGOD decomposition adds +9.4 pp; projection head adds +2.3 pp over variants without each component.
  • Localization & classification (Table 8): GUIDED raises mean classification score from 51.4%→67.8% and mean IoU from 42.5%→75.2% on LaMI-DETR.
  • FGAD VLM comparison: Refined CLIP (212 ms/image) achieves 60.8% mAP; LLaVA-1.6 achieves 51.2% but at 34,718 ms/image.
  • Inference overhead: GUIDED adds ~72 ms (LLaMA-3.1-8B) to 194 ms (LLaMA-3.3-70B) per class for LLM parsing, but this is a one-time offline cost per class name; per-image detector + CLIP overhead is ~2 ms.

Limitations

  • FGAD operates on features cropped from coarse bounding boxes; attributes that extend beyond those boxes (e.g., relational or context-dependent attributes) are missed.
  • The three-stage pipeline with a large LLM introduces per-class-name parsing latency (~72–194 ms); this is amortized for fixed class sets but real-time open-ended querying incurs this cost.
  • Subject identification accuracy degrades with small LLMs (LLaMA-3.1-8B: 9/300 errors, all propagating), requiring larger models for reliability.
  • The projection head in FGAD improves fine-grained discrimination but overfits base classes when used in CGOD (−5.8 pp), constraining its use to the discrimination stage only.

Relevance to Vision-Language Models

GUIDED directly targets a fundamental limitation of CLIP-style VLMs: contrastive pretraining entangles subject and attribute semantics in a single embedding, making those embeddings poorly suited for compositional reasoning tasks like attribute-conditioned detection. The paper provides an important diagnostic (t-SNE, localization shift, score/IoU analysis) quantifying how CLIP embeddings fail under fine-grained prompts, which is broadly relevant to anyone using CLIP for retrieval, grounding, or detection. The solution—routing coarse semantics to a detection head and fine-grained semantics to a refined VLM—is a modular template applicable to other tasks where VLM embeddings conflate multiple semantic levels. The plug-in FGAD stage also demonstrates that a lightweight projection head on frozen CLIP can recover substantial fine-grained discriminability, offering a practical technique for the broader VLM adaptation literature.