DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

DeCo-DETR decoupled cognition achieves efficient open-vocabulary detection; ICLR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeCo-DETR is a vision-centric open-vocabulary object detection (OVOD) framework that eliminates text encoder dependency at inference by precomputing a hierarchical semantic prototype space offline from LLaVA-generated region descriptions aligned via CLIP. It further decouples localization and semantic alignment through dual-stream gradient isolation, mitigating the optimization conflict that degrades novel-class generalization in prior methods.

Problem

Existing OVOD methods face two interrelated failure modes: (1) reliance on large text encoders or LLM prompting at inference time causes high latency, limiting practical deployment; (2) jointly optimizing localization and cross-modal alignment in a shared parameter space creates conflicting gradients, forcing a trade-off between closed-set detection accuracy and open-world generalization.

Method

DeCo-DETR introduces three tightly coordinated components:

Dynamic Hierarchical Concept Pool (DHCP): Offline, region proposals \(R_i\) are captioned by LLaVA to produce text \(t_i\); both \(R_i\) and \(t_i\) are embedded via CLIP and filtered by cosine similarity threshold \(\delta\). The filtered embeddings are hierarchically clustered: K-Means (\(M_1 = 1203\) centroids) for coarse prototypes, then DBSCAN within each cluster for fine-grained sub-prototypes (\(M_2 = 4800\)), yielding prototype matrix \(A \in \mathbb{R}^{d \times M}\), \(M = 6003\). During training, prototypes are updated via momentum: \(A_j \leftarrow \gamma A_j + (1-\gamma)\,\text{LayerNorm}\!\left(\sum_i D_{i,j} e_i\right)\) with \(\gamma = 0.99\).

Hierarchical Knowledge Distillation (Hi-Know DPA): Decoder queries \(q_n\) are projected into the CLIP embedding space via learnable MLP \(h_\theta\). Soft assignment weights \(w_{n,j}\) over prototypes are computed via temperature-scaled cosine similarity, and each query is enhanced as \(r_n = \sum_j w_{n,j} A_j + \text{MLP}(\hat{q}_n)\). A frozen CLIP teacher generates target distributions \(\tilde{w}_n\), and distillation is trained with \(\mathcal{L}_\text{DPA} = \mathcal{L}_\text{det} + \lambda_\text{KL}\sum_n \text{KL}(\tilde{w}_n \| w_n) + \lambda_\text{align}\mathcal{L}_\text{align}\).

Parametric Decoupling Training (PD-DuGi): Gradients from \(\mathcal{L}_\text{det}\) are restricted to the backbone/decoder; gradients from \(\mathcal{L}_\text{align} = \text{CrossEntropy}(t_n, T_\text{teacher})\) are restricted to the semantic modules \(g_\phi, h_\theta\) via StopGradient on \(q_n\). A cosine-annealed \(\lambda_\text{align}(t)\) progressively shifts training focus from localization to alignment. No text encoder is used at inference.

Key Contributions

• DHCP: Offline LLaVA+CLIP prototype construction with hierarchical clustering (K-Means + DBSCAN) and momentum-based online refinement, eliminating runtime text encoding.
• Hi-Know DPA: Attention-aware prototype aggregation for multi-granularity semantic grounding of DETR queries, supervised by frozen CLIP teacher via KL distillation.
• PD-DuGi: Dual-stream gradient isolation that cleanly separates localization and semantic alignment optimization, resolving the closed-set/open-world trade-off.
• Achieves 3.1–5.8 AP improvement over novel classes with 135 ms/image inference (R50 backbone on a single RTX 4090).

Results

OV-COCO (AP\(_{50}\), novel / base / overall): - V-OVD setting: 41.3 / 56.7 / 53.1 — vs. CAKE (38.2 / 58.0 / 52.8), DK-DETR (32.3), ViLD (29.4) - G-OVD setting: 47.1 / 60.2 / 55.0 — vs. CAKE (39.1), LP-OVOD (40.5), RALF (41.3) - C-OVD setting: 44.9 / 59.8 / 56.3 — vs. CAKE (41.3); BIRDet (46.2) still leads on novel AP - WS-OVD setting: 45.5 / 60.5 / 57.1 — vs. CAKE (41.8)

OV-LVIS (AP\(_r\) / AP\(_c\) / AP\(_f\) / AP overall): - 29.4 / 33.1 / 38.9 / 35.2 — vs. Mamba (29.3 / 34.2 / 36.8 / 35.0), CORA (22.2 / 32.0 / 40.2 / 33.5)

Inference latency (R50): 135 ms/img — comparable to Deformable DETR (120 ms) and faster than DetPro (140 ms) despite open-vocabulary capability.

Limitations

• On C-OVD, BIRDet (46.2 novel AP) still outperforms DeCo-DETR (44.9), indicating the decoupling strategy does not universally dominate.
• Offline DHCP construction requires a forward pass through LLaVA and CLIP over the training set; this preprocessing cost is not quantified in the paper.
• Ablation experiments relied on dataset cropping due to hardware constraints (8× A100), potentially limiting the statistical reliability of component-level comparisons.
• Table 3 (GFLOPs, parameter counts) is truncated in the provided text, so full compute-efficiency comparison is not verifiable.
• The prototype pool size (\(M = 6003\)) is a fixed hyperparameter; generalization to very large or fine-grained ontologies (e.g., exhaustive LVIS taxonomy) may require retuning.

Relevance to Vision-Language Models

DeCo-DETR demonstrates a practical pattern for compressing LVLM knowledge (LLaVA + CLIP) into a static prototype memory that decouples inference from online VLM calls — directly relevant to efficient VLM deployment research. The gradient-isolation training strategy addresses a fundamental tension in VLM-grounded detectors between discriminative localization and cross-modal generalization, contributing to the broader question of how to fine-tune VLM-derived representations without catastrophic forgetting of open-vocabulary knowledge. The hierarchical CLIP-aligned prototype space is a concrete instantiation of offline VLM distillation, complementing concurrent work on knowledge distillation from LVLMs into lightweight models. Researchers tracking VLMs will find this a useful case study in inference-time decoupling — the principle of precomputing text-side embeddings offline extends naturally to retrieval-augmented VLMs and vision-language retrieval systems.