Skip to content

Test-Time Adaptive Object Detection with Foundation Model

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes the first foundation-model-powered test-time adaptive object detection (TTAOD) method, building on Grounding DINO with multimodal prompt tuning and a dynamic instance memory. It eliminates dependence on source-domain statistics and breaks the closed-set assumption, enabling adaptation to arbitrary cross-domain and cross-category target data at inference time.

Problem

Existing TTAOD methods (e.g., STFAR, CTTAOD) require source-domain statistical characteristics (feature mean/variance) during adaptation, violating the source-free TTA principle. They also assume identical category spaces between source and target domains (closed-set), precluding adaptation to novel categories encountered at test time.

Method

The method builds on Grounding DINO (Swin-Tiny backbone) and has two main components:

Multi-modal Prompt-based Mean-Teacher (MPMT): Introduces learnable text prompts \(P_T\) (added to text encoder output tokens) and visual prompts \(P_{I,i}\) (prepended to image tokens at each Transformer layer \(i\)). A Mean-Teacher framework uses weak/strong augmentation pairs; only prompts are optimized; teacher prompts are updated via exponential moving average (\(\gamma = 0.999\)). A Test-time Warm-start (TTWS) initializes visual prompts by average-pooling image tokens from the first test sample (\(P_{I,i} = \text{AvgPool}(E_{I,i})\)), preventing catastrophic failure from zero/random initialization.

Instance Dynamic Memory (IDM): Maintains a per-category queue \(Q_c\) (capacity 20) of triplets (cropped image, DINOv2 feature, score) for pseudo-labels exceeding threshold \(th_{pl}=0.3\). The queue replaces the lowest-scoring entry when full. Two strategies leverage IDM: - Memory Enhancement: Computes per-category DINOv2 feature prototypes from \(Q_c\); refines prediction scores via an affinity function \(A(x) = \alpha \exp(-\beta(1-x))\) applied to cosine similarity with the prototype. Applied only to high-confidence predictions (\(th_{me}=0.3\)). - Memory Hallucination: For test images that yield no pseudo-labels, samples instances from IDM and composites them onto the negative image (up to 3 instances, random placement with IoU check, Beta-distributed mixing coefficient \(\lambda\)) to create synthetic positives for training.

Key Contributions

  • First source-free, open-vocabulary TTAOD method; no source data or closed-set assumption required.
  • Multi-modal Prompt-based Mean-Teacher framework with joint text + visual prompt tuning for parameter-efficient adaptation of a VL detector.
  • Test-time Warm-start strategy for stable visual prompt initialization.
  • Instance Dynamic Memory with Memory Enhancement (prediction re-ranking via prototype similarity) and Memory Hallucination (synthetic positive generation for pseudo-label-free batches).

Results

  • Pascal-C (AP50): Ours 56.2% vs. STFAR 45.2% (+11.0%), vs. Mean-Teacher on Grounding DINO 51.5% (+4.7%). Best across all 15 corruption types.
  • COCO-C (mAP): Ours 26.0% vs. Mean-Teacher on Grounding DINO 24.1%; state-of-the-art on 8/15 corruption types; no source data used, unlike competing methods initialized with SoftTeacher COCO-pretrained weights.
  • ODinW-13 cross-dataset (mAP): Ours 54.2% vs. Direct Test 52.8% (+1.4%); Mean-Teacher gains only +0.3%. Consistent improvement across 11/13 sub-datasets (failures on Mu/Pa with ≤5 test images).
  • Ablation (Pascal-C avg AP50): TPT alone +0.6%; VPT alone −3.4%; TPT+VPT+TTWS 53.4%; adding ME 53.9%; adding MH 56.2%.
  • Optimal visual prompt count: \(m=10\) (performance degrades beyond \(m=30\)); optimal IDM capacity: \(|Q_c|_{max}=20\).

Limitations

  • Performance degrades on extremely small test sets (Mu: 5 images, Pa: 4 images), as insufficient samples prevent effective adaptation.
  • Relies on DINOv2 (ViT-L) as a feature extractor in IDM, adding computational overhead beyond the Grounding DINO backbone.
  • Memory Hallucination requires careful tuning of capacity \(|Q_c|_{max}\); too small causes overfitting to few hallucinated images, too large admits noisy pseudo-labels.
  • Comparison with Faster R-CNN-based baselines is acknowledged as not fully fair due to detector architecture differences.
  • Hyperparameters (\(\alpha, \beta\)) differ between Pascal-C and COCO-C, suggesting sensitivity to domain type.

Relevance to Vision-Language Models

This work directly demonstrates how VLMs (specifically Grounding DINO, built on CLIP-style language grounding) can be adapted at test time via parameter-efficient prompt tuning, which is a core challenge in deploying VLMs under distribution shift. The multimodal prompt tuning strategy—combining text and visual prompts within a Mean-Teacher self-training loop—extends the prompt-learning paradigm (CoOp, VPT) to the online, streaming inference setting. For VLM researchers, the Test-time Warm-start finding highlights a previously underappreciated failure mode of visual prompt initialization unique to the test-time (no held-out validation) regime. The IDM-based pseudo-label refinement via DINOv2 prototypes also illustrates how auxiliary foundation models can compensate for VLM weaknesses under severe domain shift.