VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval¶

🕒 Published (v1): 2026-07-09 14:35 UTC · Source: Arxiv · link

Why this paper was selected

Visual tokenization + vector DB for open-vocab detection/segmentation; novel retrieval-grounded approach

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VocaDet is a training-free, sample-driven open-vocabulary detection and segmentation framework that converts visual features into discrete tokens via agglomerative clustering and stores them in a vector database for scalable retrieval-based recognition. It learns from user-provided positive/negative sample collections without model retraining, and incrementally improves as the sample repository grows. An online background filtering mechanism reduces redundant retrieval in fixed-camera deployments.

Problem¶

Existing open-vocabulary detectors (Grounding DINO, T-Rex2, DINO-X) rely on text prompts, small visual example sets, or repeated per-query feature matching, which does not scale to large, continuously expanding object repositories. There is no mechanism to learn efficiently from bulk positive/negative sample collections without retraining.

Method¶

VocaDet uses DINOv3 as a frozen feature extractor. Dense patch features are discretized into visual tokens (visual words) via agglomerative clustering with an adaptive sensitivity parameter that controls granularity. Multiple clustering sensitivities produce multi-granularity token sets. Position-debiased representations (following INSID3) and spatial topology (adjacency lists between co-occurring clusters within an annotated region) are stored in a Milvus vector database. Annotations support points, boxes, or masks; a cluster is stored if >90% of its area falls within the annotated region.

At inference, the query image is tokenized identically and each visual word is matched against the database. A detection is accepted only when both feature similarity exceeds a threshold and topological adjacency between matched clusters mirrors what is stored (topology-consistent retrieval). Spatial clusters of matched tokens are converted to bounding boxes, then NMS merges multi-granularity overlapping boxes.

A background filtering layer maintains an online background memory: visual words matching stored background vectors are excluded from database retrieval before object matching. The memory is updated per frame via a match-and-replace strategy, adapting to scene changes.

Key Contributions¶

Training-free, sample-driven open-vocabulary detection/segmentation that scales to large positive/negative repositories without retraining.
Discrete visual vocabulary via adaptive agglomerative clustering on DINOv3 features, with multi-granularity token hierarchies.
Topology-aware vector database storage (cluster adjacency encoded in indexList fields) enabling spatially consistent multi-cluster object matching.
Online background filtering mechanism for practical fixed-camera scenarios that eliminates redundant retrieval of static background tokens.
Continuously expandable recognition: performance improves incrementally as more annotated samples are added to the database.

Results¶

Evaluated on UA-DETRAC (vehicle detection benchmark).
VocaDet achieves "effective open-vocabulary detection performance without conventional detector training" and demonstrates "competitive detection performance."
No quantitative numbers (AP, mAP, FPS, etc.) are reported in the text; the paper characterizes results qualitatively as competitive and notes continuous improvement with sample accumulation.

Limitations¶

Instance merging failure: Spatially adjacent objects of the same category with similar appearance may be merged into a single detection, because the method lacks discriminative power to separate neighboring instances relying solely on visual similarity and spatial clustering.
Cold-start problem: Initial database construction with few samples makes it difficult to accurately characterize category distributions and establish reliable feature boundaries.
No ablation or quantitative comparison against named baselines is presented in the provided text.
Scalability and latency at very large database sizes are not characterized.

Relevance to Vision-Language Models¶

VocaDet represents a vision-only counterpart to VLM-based open-vocabulary detectors (Grounding DINO, T-Rex2), replacing language-grounded recognition with purely visual token retrieval. It is relevant to the VLM tracking thread because it isolates the role of visual representations (DINOv3) from language, showing how far retrieval-based methods can go without text supervision. The framework challenges the assumption that open-vocabulary perception requires language alignment, and could serve as a complementary or diagnostic baseline for VLM-grounded detection systems. The topology-aware matching mechanism is architecturally distinct from cross-attention-based visual grounding used in most VLMs.